wehub/maris205--dnagpt2
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:37:17 +08:00
commit dbd2894833
201 changed files with 3431374 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
01-data_env/.ipynb_checkpoints/1-env-intro-checkpoint.ipynb
+79
View File
@@ -0,0 +1,79 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "a25f3d36-e14f-4afd-8926-32748a42e1d1",
"metadata": {},
"source": [
"# 1 大模型运行环境简介\n",
"\n",
"\n",
"建议直接使用autodl，google colab等环境\n",
"\n",
"显卡：4090或者4090d\n",
"\n",
"内存：32G至少\n",
"\n",
"torch>=2.3.0\n",
"\n",
"具体可以参考：https://zhuanlan.zhihu.com/p/13479003076\n",
"\n",
"pip安装下面的基本transformer环境即可："
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cdeae2e5-2a39-4370-a5ec-47780f8fa76a",
"metadata": {},
"outputs": [],
"source": [
"!pip install transformers sentencepiece google protobuf deepspeed peft datasets "
]
},
{
"cell_type": "markdown",
"id": "a355a6e6-62fc-4b8f-ba35-b9c2f0ef48c8",
"metadata": {},
"source": [
"如要运行deepspeed，一般使用一机多卡即可，本教程一般不会涉及需要多机多卡的案例\n",
"\n",
"\n",
"推荐的gpu主机：\n",
"* autodl.com, 国内的 \n",
"* vast.ai, 海外的\n",
"\n",
"主流云平台gpu一般都特别贵，也不允许运行4090等显卡。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "813c37df-9fef-453a-bfbd-46dd05ac76dd",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
01-data_env/.ipynb_checkpoints/2-data-intro-checkpoint.ipynb
+283
View File
@@ -0,0 +1,283 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "50ff8836-7075-4858-b463-c99f973f408d",
"metadata": {},
"source": [
"# 2 基因相关预训练和微调数据"
]
},
{
"cell_type": "markdown",
"id": "17cde5bb-70e5-437e-a4a3-193a881dd412",
"metadata": {},
"source": [
"本教程主要关注基因相关的生物序列数据，包括主要的DNA和蛋白质序列，data目录下数据如下：\n",
"\n",
"* dna_1g.txt DNA序列数据，大小1G，从glue数据集中抽取，具体可参考dnabert2的论文，包括多个模式生物的数据\n",
"* potein_1g.txt 蛋白质序列数据，大小1G，从pdb数据库中抽取\n",
"* english_500m.txt 英文数据，大小500M，就是英文百科"
]
},
{
"cell_type": "markdown",
"id": "b45ecf27-1514-45e0-bfbd-361e6dcc98ea",
"metadata": {},
"source": [
"下面演示下huggingface的dataset库的基本用法，以及样例数据"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2715f9bb-2e43-4bd6-8715-5c96d317bcf8",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c067aeb8ab304723ac6b527e7ad6c768",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0 examples [00:00, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['text'],\n",
" num_rows: 1079595\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#读取dna数据\n",
"from datasets import load_dataset\n",
"dna_dataset = load_dataset('text', data_files='data/dna_1g.txt')\n",
"dna_dataset"
]
},
{
"cell_type": "markdown",
"id": "ec00ad72-c5f9-40db-8508-6c6bf8f374c1",
"metadata": {},
"source": [
"\n",
"Datasets 提供了加载脚本来加载本地和远程数据集。它支持几种常见的数据格式，例如：\n",
"\n",
"| Data format | Loading script | Example |\n",
"|-------------------|----------------|-------------------------------------------------------------------------|\n",
"| CSV & TSV | csv | `load_dataset(\"csv\", data_files=\"my_file.csv\")` |\n",
"| Text files | text | `load_dataset(\"text\", data_files=\"my_file.txt\")` |\n",
"| JSON & JSON Lines | json | `load_dataset(\"json\", data_files=\"my_file.jsonl\")` |\n",
"| Pickled DataFrames| pandas | `load_dataset(\"pandas\", data_files=\"my_dataframe.pkl\")` |\n",
"\n",
"如表所示, 对于每种数据格式, 我们只需要使用 load_dataset() 函数, 使用 data_files 指定一个或多个文件的路径的参数。 "
]
},
{
"cell_type": "markdown",
"id": "24c40ec7-cb59-4c3a-8052-00d7979f6208",
"metadata": {},
"source": [
"load_dataset默认加载到train下，可以把dataset当做一个一般的python dict使用"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "2a375409-d2b6-4648-8f6a-8ac3fb25bb75",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': 'TTAAATCCTAGAAGTTGGTTACACGGGTGAGGAAAATGGTGAGAAGCCCAATGGGATGCTGTAGCAATGACAGTGAACTGCTGTCACCCCTGAGGCTGGAAAGATAACAGACATTTGCCAGGAGCTAGAAGCTGGGGCAGCCTGGTAGGAGCGAGAATATGGTGAGAGCTGCCCCCTGGGGATGGAACCACAGAGGGAGGGTCTCTCTGATGAGACATAGAGCCAAGAACAGATACAGCCATTGTGGGAGATGGTAACCAAAGCAGAGAGAGAGAGAGAGAGCGAGAGAGAGAGAAAACACCCTGGTTTCTTCCTTCCTTCCACCTTTGAGTTTCCCACCAGTGCTTCCCATTAGCCCAAACTACCAAGAACCCAGAGGGCAAAGGAGCCCGGGAAATCTAATTCTACATGATACCGAGCAAAGCCGATGTTCCAGCTGGCTGCGTCTGTTACAGTAGGTAGTCAGGCAGACATAAGCAGGGCAGGAGAGGGCTCCTCCCAACCAGGAATGTCAGGTGACGGTCAGGTGATGGTCAGGTGGTCATTAACTGTCTCTCTAAAATAATAATTGGTTACAGCCAGCACCAGGGAAAGGCAGTCTCCCAACCGATAGAAACATCTGAAACTGATGATCAGTAGCTTCCCAATAAGGTCTCAGGAGTTGGACGCATGGGCTCAGCATGAACACTGAGAGGCAAAATGGTGGAGTTTAACTGGTATATGACCTTCCTCTAGAAACATTCAGCTGGTAAGGGAAGAACGCCTTAAGCGAATATGCACGCAACTCCAGTAAACACTGTGCATGTTCCTGTCCCAATGCTGGTAGACCACTGCGCATGCAAACAGCCCACCCCAGGGAAGAATCAGGAGAGAAGAGACCCCACAAGCATGCCAACACATAAAACCCCAAGTCAGGAGTCAAACCATGCACTTGAATCAAGTCACCCACTTAGCTCTCTTTCAAGTGTATTTTACTTTCTTTCATTCCTGCTCTAAAACT'}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dna_dataset[\"train\"][0]"
]
},
{
"cell_type": "raw",
"id": "985bd82a-1ff0-49ef-968d-8d5f6df8d76f",
"metadata": {},
"source": [
"dna数据就是如上所示，由ATCG 4个字母组成的文本，对于学习大语言模型而言，可以不关注其具体的含义，当然，大部分dna序列的含义目前也都没有解读：）\n",
"\n",
"然后是蛋白质序列"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "94e3f443-939e-4148-bba6-6cafa90790b6",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a1023bd5311a4a5dbe96c6c8fdc5b519",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0 examples [00:00, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'text': 'MLTDPFGRTIKLRIAVTRCLCIYCHREGESDPGTEMSAERIAEIAKAFYELGIKKLKLTGGEPLLRKDICEIISMMPDFEEISLTTGILLSDLAFDLKESGLDRVISLDTLDAETFRFITGGGELSRVLEGLRMAVEAKLTPIKLMVLMSGLESEVRKMLEFASFEETVILQLIELIPSRTGKFYLDPTIFEKDFERVAKAVKIRDMHRRKQFITPFGVVEIVKPLDTEFCMHCRIRITSDGRIKLCLMSDETVDISELSGDELKKAIFEAVKRRKPFFIMKGEILALISAVLWGFAPILDRYALLSGAPIYAALAIRAFGALIAMLFILSVLRGGLAVEAKAAVLLLIAGAIGGALAMVFYYLALESVGASRTVPITAIYPMFTALFSFLLLSEPLSPKTIAGIAFIVLGVILVSEGMVKLRGEDVVIRKYDHSMDRDKLIEMYVYDPRFRCLGLPPLSKEAIKGWIDYLGQGFAIIAEKDGKIVGHLVIVPGEREVDLTIFIHQDYQLGLGQEMMKLIIDFCRKAGFAITLVTERTARAIHVYRKLGFEIVAPYYEYDMRLQLKMIVPKGKTVLIKGTASIRGECEVLGARLFFESEKFVPVFCLEDCEIEVGEFKILDGSTIPESWEKLSKMDWETVFLYGGVDSGKSTLATYLAKVGGAYVLDLDIGQADVAPGAMGYGFAKDVVSLSKVSMIGFFVGSITPQGREAKCLRGVARLWKELRKLDGRKIIDTTGWVRGRRAKEYKLAKLEIIEPDLIASFEGKLFDWKTFEVEKGYVIRRDKDRAKARFESYRKFLDGAKTFELERDGIKLKPDFFKGKDVSQFIESVLGTRVVFARLGEEHLTICTKEDCPEYEILRELKELYEVDDIFLFSESEARFVAGLYRGKKYLGIGLIKSIDRILLECTQSDFDTIEIGEIRLEDGRECFIKRFMARIAYSYKPQDETRAARAMGYEVPISFKHAMEICRVLKGKKVPQAISFLEEVVQLKVPVPFRKHKKKVAHKIPGWYAGRYPQKAAEILKVLKLKAAEYKGLKAEELIIVHAQAKK'}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"protein_dataset = load_dataset('text', data_files='data/protein_1g.txt')\n",
"protein_dataset[\"train\"][0]"
]
},
{
"cell_type": "markdown",
"id": "ecaa8216-7b9f-4ba0-af8e-c7c868dc7ec9",
"metadata": {},
"source": [
"蛋白质序列，则是有MLTDP等20个字母/氨基酸 组成的文本，当然，目前对蛋白质的理解远超过对DNA的。\n",
"\n",
"然后就是英文文本了，这个就比较容易看懂"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7521f8ea-fd70-4f5b-aeeb-7ff81635320d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': ' \" There \\'s Got to Be a Way \" is a song by American singer and songwriter Mariah Carey from her self @-@ titled debut studio album ( 1990 ) . Columbia released it as the fifth and final single from the album in the United Kingdom . It was one of four songs Carey wrote with Ric Wake during their first recording session together , but \" There \\'s Got to Be a Way \" was the only composition to make the final track listing . It is a socio @-@ political conscious R & B @-@ pop song which addresses the existence of poverty , racism and war in the world which gradually becomes more aspirational and positive as it progresses . The track garnered a mixed reception upon the album \\'s release in 1990 . While Carey \\'s vocals were praised , it was seen as too political . An accompanying music video highlights social injustices . The song reached number 54 on the UK Singles Chart . '}"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"english_dataset = load_dataset('text', data_files='data/english_500m.txt')\n",
"english_dataset[\"train\"][301]"
]
},
{
"cell_type": "markdown",
"id": "5fcad08d-6389-453e-997f-eb2877a5fbbb",
"metadata": {},
"source": [
"英文序列，就是26个字母组成的文本了，当然，英文是包括空格的，生物序列则没有明确的空格"
]
},
{
"cell_type": "markdown",
"id": "5e4e1e85-a187-469d-9950-1c6cbb9c41f7",
"metadata": {},
"source": [
"前面这些数据集，就是常规的文本，一般就是当做预训练数据使用，而分类等下游微调任务，一般都是包含标签的，多写成json或者csv的格式，这里也给出一个例子："
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "c48dd04e-af42-4222-94d5-56a8e08e2cbf",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7c611d1ab3bb408394196e7929d8e0c5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0 examples [00:00, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'sentence1': 'ATGGAGGAAAATCAGACCATGGTCACAGAGTTCGTCCTGCTGGGATTCTGTCTTGGCCCGAGGATTCACCTAGTTCTTTTTCTGCTTTTCTCTCTCTTCTATACTCTCACCATACTGGGGAATGGGACTATCCTTGCAATGATCTGCCTGGACTCCAGACTCCACACTCCCATGTACTTCTTCCTGTCCCACCTGGCCATTGTCGATATGGCCTATGCCTGCAACACAGTGCCTCAGACACTCATAAACCTCTTGGATGAGACCAGGCCCATCACCTTTGCTGGATGCATGACACAGACCTTTCTCTTCTTGGCTTTTGCCCACACTGAATGTGTGCTCCTTGTTGTGATGTCCTATGACCGGTATGTAGCTATCTGCCACCCGCTACACTACACTGTCATCATGAACTGGAGAGTGTGTACCATTCTGGCTGCTGTTTCCTGGATATTTAGCTTTCTCCTTGCTCTGGTCCATTTAGTTCTCATCCTGAGGCTGCCCTTCTGTGGACCTCATGAAATCAATCACTTCTTCTGTGAAATCCTGTCTGTCCTCAAGCTGGCCTGTGCTGACACAACACTCAATCAGGTCGTTATCTTTGCAGCTTGTGTGTTCATATTAGTGGCCCCCCTATGCTTTGTACTAGTCTCCTACACACGCATCCTGGTGGCCATCCTGAGGATCCAGTCAGGGGAGGGACGCAGAAAGGCCTTCTCTACCTGTTCCTCCCACCTCTGTGTGGTAGGGCTCTTCTTTGGCAGTGCCATTGTCATGTACATGGCCCCCAAGTCCCAGCACCCAGAGGAGCAGCAGAAGGTTCTTTTCCTGTTTTACAGTTTTTTCAACCCCATGCTGAACCCCCTAATCTACAGTCTGAGGAATGCTGAGGTGAAGGGCGCCCTCAAGAGGTCACTGTGCAAAGAAAGTCATTCCTGGTTGGTGTGGTGTTCGGACCATAAATCTTGG',\n",
" 'sentence2': 'MEENQTMVTEFVLLGFCLGPRIHLVLFLLFSLFYTLTILGNGTILAMICLDSRLHTPMYFFLSHLAIVDMAYACNTVPQTLINLLDETRPITFAGCMTQTFLFLAFAHTECVLLVVMSYDRYVAICHPLHYTVIMNWRVCTILAAVSWIFSFLLALVHLVLILRLPFCGPHEINHFFCEILSVLKLACADTTLNQVVIFAACVFILVAPLCFVLVSYTRILVAILRIQSGEGRRKAFSTCSSHLCVVGLFFGSAIVMYMAPKSQHPEEQQKVLFLFYSFFNPMLNPLIYSLRNAEVKGALKRSLCKESHSWLVWCSDHKSW',\n",
" 'label': 1}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ft_dataset = load_dataset('json', data_files='data/dna_protein_my.json')\n",
"ft_dataset[\"train\"][0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8f3ec639-e426-4233-a20a-dad94069175b",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
01-data_env/.ipynb_checkpoints/3-dataset-use-checkpoint.ipynb
+259
View File
@@ -0,0 +1,259 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "68c06a52-e27c-4da6-8a02-cd010270bedf",
"metadata": {},
"source": [
"# 3 datasets库基本使用"
]
},
{
"cell_type": "markdown",
"id": "2dc4c70f-694c-4785-81d8-26ebab2b7210",
"metadata": {},
"source": [
"## 基本使用\n",
"上一节中，已经介绍了使用datasets读取本地文件的方法，这一节继续介绍datasets一些常用的方法\n",
"\n",
"首先是数据分割，因为我们从数据源获得DNA序列等数据，都是一个文本文件，但训练的时候，一般都需要分成训练集和测试集等\n",
"\n",
"一个简单的例子如下所示："
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "6e9f346f-31f6-40cc-86e5-723c65033883",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['text'],\n",
" num_rows: 1025615\n",
" })\n",
" test: Dataset({\n",
" features: ['text'],\n",
" num_rows: 53980\n",
" })\n",
"})"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#读取dna数据\n",
"from datasets import load_dataset\n",
"dna_dataset = load_dataset('text', data_files='data/dna_1g.txt')['train'].rain_test_split(test_size=0.05) #默认已经shuffle\n",
"dna_dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "75900650-74da-4ca9-a285-b2832a5a1485",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': 'ATGTGTGCAATGGGTTATCTTTATGTAATAACAGTCATATCACGGGTGTTCCTCAGAAGTAGTGAACTGGCTAGCATTTTTAGACACTATGTGATCTCTCATATGACTACACTCAATTTAAAATAAAATGAAATGTGTTGTGTGTGTCTAAAATCTATAAAGGGAAAAGTATCTTAAGTATTTTTTAGATGTTAAAGTAGATGTGTATCCTAAAATATGCATTGTTCACAGATGTTAAAATTACAACTACAATCTGTGAAACACAGATCTTAGGACAGCAATGTTTCACAAGAAAAAAAATGATGCAGCCTTCTTTAGTATTTATAGTCATTTGAACAATTATGGCAACCATAAGTTCATATATAACATCCCCATTTGGTGAAACTAGTTGGGAAAGATTAGAAGGTATGACCTTGTTGGAGGAACTATACCATTGGGGTGGCTTTGAGACTTCAGAAGTTTCAAGGCCCATTTAGTGCTTTCTACCTTATGAAGCTGTGAGTTCTCCTTGCTAGCTACATAACTTGGAAAGCAGGCCCTGCACTTCACCCAAGGAGCACATTAGAGCTGGCCCTTTTGGAAGGCAATTGCGTAAGCCACACCAGGGCACCAGAGATCTGGCACTGCCATGCTCCTGCTTGCAAGTAGTGGTGTGGGTGTTGGGTGATGCCCTCCAGTCCCACCTTTTGCCACCTGTAGTAGTCAGGGGAGTTGGCCTAAGGGCATGAGAGCCTAAGACTTCACCCTAATCCCTCACCAACTGTAGCATGTGGAAGAGCAGGCTCTGTACCTTCCCTGGGCAACACATTGGAGCTGGCCCCTCACAGGCTGCAGGACTTGGGAGAGTGAGTGCTGCACCTTGACTGTGAAGGTGGTTTTGGAGGTGTGGGTGTGAGACCATGAGACCAAGAGAGGAATGGAATATTACTCACTTATTAAAAACAATGACTTCATGAAATTTGCAGGCAAATGGATGGAACTTGAAAATATCCTGAGTGAG'}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dna_dataset[\"test\"][0]"
]
},
{
"cell_type": "markdown",
"id": "cdcc5404-6590-47a4-be2c-2c1d35d3bae4",
"metadata": {},
"source": [
"可以看到，数据集已经分割成了train和test两个数据集，而在分割的时候，已经进行的随机处理\n",
"\n",
"当然，如果数据集过大，我们只需要其中一部分，这个也是一个常见的需求，一般可以使用 Dataset.select() 函数"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "049ad194-cb60-4b0f-8554-1915bfc7a9cd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['text'],\n",
" num_rows: 50000\n",
" })\n",
" valid: Dataset({\n",
" features: ['text'],\n",
" num_rows: 500\n",
" })\n",
"})"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset, DatasetDict\n",
"\n",
"dna_dataset_sample = DatasetDict(\n",
" {\n",
" \"train\": dna_dataset[\"train\"].shuffle().select(range(50000)), \n",
" \"valid\": dna_dataset[\"test\"].shuffle().select(range(500))\n",
" }\n",
")\n",
"dna_dataset_sample"
]
},
{
"cell_type": "markdown",
"id": "50cceda3-36ca-4fa6-bfb5-1dbeb155fe4f",
"metadata": {},
"source": [
"可以看到，我们使用DatasetDict来直接构造datasets，先使用shuffle()来随机，然后使用select来选择前n个数据\n",
"\n",
"select的参数为indices (list 或 range): 索引列表或范围对象，指明要选择哪些样本，如dataset.select([0, 2, 4])就是选择1,3,5条记录"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "17a1fa7c-ff4b-419f-8a82-e58cc5777cd4",
"metadata": {},
"source": [
"## 读取线上库\n",
"\n",
"当然，数据也可以直接从huggingface的线上仓库读取，这时候需要注意科学上网问题。\n",
"\n",
"具体使用函数也是load_dataset\n",
"\n",
"<img src='img/datasets_dnagpt.png' width='800px' />"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "6ae24950-2c74-457b-b1f2-d2e4397e1fa1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"#或者\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "30ff9798-d06d-4992-81fc-03102f03599b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 59196\n",
" })\n",
"})"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset\n",
"dna_data = load_dataset(\"dnagpt/dna_core_promoter\")\n",
"dna_data"
]
},
{
"cell_type": "markdown",
"id": "30c4b754-af11-4ac1-9742-45427059617e",
"metadata": {},
"source": [
"当然，如果你想分享你的数据集到huggingface上面，也是一行函数即可："
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f9847be9-e085-41e3-ad29-a450cc017d64",
"metadata": {},
"outputs": [],
"source": [
"dna_data.push_to_hub(\"org_name/your_dataset_name\", token=\"hf_yourtoken\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
01-data_env/1-env-intro.ipynb
+130
View File
@@ -0,0 +1,130 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "a25f3d36-e14f-4afd-8926-32748a42e1d1",
"metadata": {},
"source": [
"# 1 大模型运行环境简介\n",
"\n",
"\n",
"建议直接使用autodl，google colab等环境\n",
"\n",
"显卡：4090或者4090d\n",
"\n",
"内存：32G至少\n",
"\n",
"torch>=2.3.0\n",
"\n",
"具体可以参考：https://zhuanlan.zhihu.com/p/13479003076\n",
"\n",
"pip安装下面的基本transformer环境即可："
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cdeae2e5-2a39-4370-a5ec-47780f8fa76a",
"metadata": {},
"outputs": [],
"source": [
"!pip install transformers sentencepiece google protobuf deepspeed peft datasets "
]
},
{
"cell_type": "markdown",
"id": "a355a6e6-62fc-4b8f-ba35-b9c2f0ef48c8",
"metadata": {},
"source": [
"如要运行deepspeed，一般使用一机多卡即可，本教程一般不会涉及需要多机多卡的案例\n",
"\n",
"\n",
"推荐的gpu主机：\n",
"* autodl.com, 国内的 \n",
"* vast.ai, 海外的\n",
"\n",
"主流云平台gpu一般都特别贵，也不允许运行4090等显卡。"
]
},
{
"cell_type": "markdown",
"id": "4ee0674c-f001-453f-b0b0-7e3b25309040",
"metadata": {},
"source": [
"另外，建议把jupyter的注释打开，这样非常方便学习\n",
"\n",
"<img src=\"img/zhushi.png\" alt=\"示例图片\" width=\"500px\" />"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "444adc87-78c8-4209-8260-0c5c4a668ea0",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06d9dc67-dbd4-4d37-bbdb-ccf59c8fdbf9",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2168e365-8254-4063-98bd-27afdbdb2f32",
"metadata": {},
"outputs": [],
"source": [
"#lfs 支持\n",
"!apt-get update\n",
"\n",
"!apt-get install git-lfs\n",
"\n",
"!git lfs install"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
01-data_env/2-data-intro.ipynb
+283
View File
@@ -0,0 +1,283 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "50ff8836-7075-4858-b463-c99f973f408d",
"metadata": {},
"source": [
"# 2 基因相关预训练和微调数据"
]
},
{
"cell_type": "markdown",
"id": "17cde5bb-70e5-437e-a4a3-193a881dd412",
"metadata": {},
"source": [
"本教程主要关注基因相关的生物序列数据，包括主要的DNA和蛋白质序列，data目录下数据如下：\n",
"\n",
"* dna_1g.txt DNA序列数据，大小1G，从GUE数据集中抽取，具体可参考dnabert2的论文，包括多个模式生物的数据(https://github.com/MAGICS-LAB/DNABERT_2)\n",
"* potein_1g.txt 蛋白质序列数据，大小1G，从pdb/uniprot数据库中抽取(https://www.uniprot.org/help/downloads)\n",
"* english_500m.txt 英文数据，大小500M，就是英文百科(https://huggingface.co/datasets/Salesforce/wikitext, https://huggingface.co/datasets/iohadrubin/wikitext-103-raw-v1)"
]
},
{
"cell_type": "markdown",
"id": "b45ecf27-1514-45e0-bfbd-361e6dcc98ea",
"metadata": {},
"source": [
"下面演示下huggingface的dataset库的基本用法，以及样例数据"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2715f9bb-2e43-4bd6-8715-5c96d317bcf8",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c067aeb8ab304723ac6b527e7ad6c768",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0 examples [00:00, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['text'],\n",
" num_rows: 1079595\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#读取dna数据\n",
"from datasets import load_dataset\n",
"dna_dataset = load_dataset('text', data_files='data/dna_1g.txt')\n",
"dna_dataset"
]
},
{
"cell_type": "markdown",
"id": "ec00ad72-c5f9-40db-8508-6c6bf8f374c1",
"metadata": {},
"source": [
"\n",
"Datasets 提供了加载脚本来加载本地和远程数据集。它支持几种常见的数据格式，例如：\n",
"\n",
"| Data format | Loading script | Example |\n",
"|-------------------|----------------|-------------------------------------------------------------------------|\n",
"| CSV & TSV | csv | `load_dataset(\"csv\", data_files=\"my_file.csv\")` |\n",
"| Text files | text | `load_dataset(\"text\", data_files=\"my_file.txt\")` |\n",
"| JSON & JSON Lines | json | `load_dataset(\"json\", data_files=\"my_file.jsonl\")` |\n",
"| Pickled DataFrames| pandas | `load_dataset(\"pandas\", data_files=\"my_dataframe.pkl\")` |\n",
"\n",
"如表所示, 对于每种数据格式, 我们只需要使用 load_dataset() 函数, 使用 data_files 指定一个或多个文件的路径的参数。 "
]
},
{
"cell_type": "markdown",
"id": "24c40ec7-cb59-4c3a-8052-00d7979f6208",
"metadata": {},
"source": [
"load_dataset默认加载到train下，可以把dataset当做一个一般的python dict使用"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "2a375409-d2b6-4648-8f6a-8ac3fb25bb75",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': 'TTAAATCCTAGAAGTTGGTTACACGGGTGAGGAAAATGGTGAGAAGCCCAATGGGATGCTGTAGCAATGACAGTGAACTGCTGTCACCCCTGAGGCTGGAAAGATAACAGACATTTGCCAGGAGCTAGAAGCTGGGGCAGCCTGGTAGGAGCGAGAATATGGTGAGAGCTGCCCCCTGGGGATGGAACCACAGAGGGAGGGTCTCTCTGATGAGACATAGAGCCAAGAACAGATACAGCCATTGTGGGAGATGGTAACCAAAGCAGAGAGAGAGAGAGAGAGCGAGAGAGAGAGAAAACACCCTGGTTTCTTCCTTCCTTCCACCTTTGAGTTTCCCACCAGTGCTTCCCATTAGCCCAAACTACCAAGAACCCAGAGGGCAAAGGAGCCCGGGAAATCTAATTCTACATGATACCGAGCAAAGCCGATGTTCCAGCTGGCTGCGTCTGTTACAGTAGGTAGTCAGGCAGACATAAGCAGGGCAGGAGAGGGCTCCTCCCAACCAGGAATGTCAGGTGACGGTCAGGTGATGGTCAGGTGGTCATTAACTGTCTCTCTAAAATAATAATTGGTTACAGCCAGCACCAGGGAAAGGCAGTCTCCCAACCGATAGAAACATCTGAAACTGATGATCAGTAGCTTCCCAATAAGGTCTCAGGAGTTGGACGCATGGGCTCAGCATGAACACTGAGAGGCAAAATGGTGGAGTTTAACTGGTATATGACCTTCCTCTAGAAACATTCAGCTGGTAAGGGAAGAACGCCTTAAGCGAATATGCACGCAACTCCAGTAAACACTGTGCATGTTCCTGTCCCAATGCTGGTAGACCACTGCGCATGCAAACAGCCCACCCCAGGGAAGAATCAGGAGAGAAGAGACCCCACAAGCATGCCAACACATAAAACCCCAAGTCAGGAGTCAAACCATGCACTTGAATCAAGTCACCCACTTAGCTCTCTTTCAAGTGTATTTTACTTTCTTTCATTCCTGCTCTAAAACT'}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dna_dataset[\"train\"][0]"
]
},
{
"cell_type": "raw",
"id": "985bd82a-1ff0-49ef-968d-8d5f6df8d76f",
"metadata": {},
"source": [
"dna数据就是如上所示，由ATCG 4个字母组成的文本，对于学习大语言模型而言，可以不关注其具体的含义，当然，大部分dna序列的含义目前也都没有解读：）\n",
"\n",
"然后是蛋白质序列"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "94e3f443-939e-4148-bba6-6cafa90790b6",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a1023bd5311a4a5dbe96c6c8fdc5b519",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0 examples [00:00, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'text': 'MLTDPFGRTIKLRIAVTRCLCIYCHREGESDPGTEMSAERIAEIAKAFYELGIKKLKLTGGEPLLRKDICEIISMMPDFEEISLTTGILLSDLAFDLKESGLDRVISLDTLDAETFRFITGGGELSRVLEGLRMAVEAKLTPIKLMVLMSGLESEVRKMLEFASFEETVILQLIELIPSRTGKFYLDPTIFEKDFERVAKAVKIRDMHRRKQFITPFGVVEIVKPLDTEFCMHCRIRITSDGRIKLCLMSDETVDISELSGDELKKAIFEAVKRRKPFFIMKGEILALISAVLWGFAPILDRYALLSGAPIYAALAIRAFGALIAMLFILSVLRGGLAVEAKAAVLLLIAGAIGGALAMVFYYLALESVGASRTVPITAIYPMFTALFSFLLLSEPLSPKTIAGIAFIVLGVILVSEGMVKLRGEDVVIRKYDHSMDRDKLIEMYVYDPRFRCLGLPPLSKEAIKGWIDYLGQGFAIIAEKDGKIVGHLVIVPGEREVDLTIFIHQDYQLGLGQEMMKLIIDFCRKAGFAITLVTERTARAIHVYRKLGFEIVAPYYEYDMRLQLKMIVPKGKTVLIKGTASIRGECEVLGARLFFESEKFVPVFCLEDCEIEVGEFKILDGSTIPESWEKLSKMDWETVFLYGGVDSGKSTLATYLAKVGGAYVLDLDIGQADVAPGAMGYGFAKDVVSLSKVSMIGFFVGSITPQGREAKCLRGVARLWKELRKLDGRKIIDTTGWVRGRRAKEYKLAKLEIIEPDLIASFEGKLFDWKTFEVEKGYVIRRDKDRAKARFESYRKFLDGAKTFELERDGIKLKPDFFKGKDVSQFIESVLGTRVVFARLGEEHLTICTKEDCPEYEILRELKELYEVDDIFLFSESEARFVAGLYRGKKYLGIGLIKSIDRILLECTQSDFDTIEIGEIRLEDGRECFIKRFMARIAYSYKPQDETRAARAMGYEVPISFKHAMEICRVLKGKKVPQAISFLEEVVQLKVPVPFRKHKKKVAHKIPGWYAGRYPQKAAEILKVLKLKAAEYKGLKAEELIIVHAQAKK'}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"protein_dataset = load_dataset('text', data_files='data/protein_1g.txt')\n",
"protein_dataset[\"train\"][0]"
]
},
{
"cell_type": "markdown",
"id": "ecaa8216-7b9f-4ba0-af8e-c7c868dc7ec9",
"metadata": {},
"source": [
"蛋白质序列，则是有MLTDP等20个字母/氨基酸 组成的文本，当然，目前对蛋白质的理解远超过对DNA的。\n",
"\n",
"然后就是英文文本了，这个就比较容易看懂"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7521f8ea-fd70-4f5b-aeeb-7ff81635320d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': ' \" There \\'s Got to Be a Way \" is a song by American singer and songwriter Mariah Carey from her self @-@ titled debut studio album ( 1990 ) . Columbia released it as the fifth and final single from the album in the United Kingdom . It was one of four songs Carey wrote with Ric Wake during their first recording session together , but \" There \\'s Got to Be a Way \" was the only composition to make the final track listing . It is a socio @-@ political conscious R & B @-@ pop song which addresses the existence of poverty , racism and war in the world which gradually becomes more aspirational and positive as it progresses . The track garnered a mixed reception upon the album \\'s release in 1990 . While Carey \\'s vocals were praised , it was seen as too political . An accompanying music video highlights social injustices . The song reached number 54 on the UK Singles Chart . '}"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"english_dataset = load_dataset('text', data_files='data/english_500m.txt')\n",
"english_dataset[\"train\"][301]"
]
},
{
"cell_type": "markdown",
"id": "5fcad08d-6389-453e-997f-eb2877a5fbbb",
"metadata": {},
"source": [
"英文序列，就是26个字母组成的文本了，当然，英文是包括空格的，生物序列则没有明确的空格"
]
},
{
"cell_type": "markdown",
"id": "5e4e1e85-a187-469d-9950-1c6cbb9c41f7",
"metadata": {},
"source": [
"前面这些数据集，就是常规的文本，一般就是当做预训练数据使用，而分类等下游微调任务，一般都是包含标签的，多写成json或者csv的格式，这里也给出一个例子："
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "c48dd04e-af42-4222-94d5-56a8e08e2cbf",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7c611d1ab3bb408394196e7929d8e0c5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0 examples [00:00, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'sentence1': 'ATGGAGGAAAATCAGACCATGGTCACAGAGTTCGTCCTGCTGGGATTCTGTCTTGGCCCGAGGATTCACCTAGTTCTTTTTCTGCTTTTCTCTCTCTTCTATACTCTCACCATACTGGGGAATGGGACTATCCTTGCAATGATCTGCCTGGACTCCAGACTCCACACTCCCATGTACTTCTTCCTGTCCCACCTGGCCATTGTCGATATGGCCTATGCCTGCAACACAGTGCCTCAGACACTCATAAACCTCTTGGATGAGACCAGGCCCATCACCTTTGCTGGATGCATGACACAGACCTTTCTCTTCTTGGCTTTTGCCCACACTGAATGTGTGCTCCTTGTTGTGATGTCCTATGACCGGTATGTAGCTATCTGCCACCCGCTACACTACACTGTCATCATGAACTGGAGAGTGTGTACCATTCTGGCTGCTGTTTCCTGGATATTTAGCTTTCTCCTTGCTCTGGTCCATTTAGTTCTCATCCTGAGGCTGCCCTTCTGTGGACCTCATGAAATCAATCACTTCTTCTGTGAAATCCTGTCTGTCCTCAAGCTGGCCTGTGCTGACACAACACTCAATCAGGTCGTTATCTTTGCAGCTTGTGTGTTCATATTAGTGGCCCCCCTATGCTTTGTACTAGTCTCCTACACACGCATCCTGGTGGCCATCCTGAGGATCCAGTCAGGGGAGGGACGCAGAAAGGCCTTCTCTACCTGTTCCTCCCACCTCTGTGTGGTAGGGCTCTTCTTTGGCAGTGCCATTGTCATGTACATGGCCCCCAAGTCCCAGCACCCAGAGGAGCAGCAGAAGGTTCTTTTCCTGTTTTACAGTTTTTTCAACCCCATGCTGAACCCCCTAATCTACAGTCTGAGGAATGCTGAGGTGAAGGGCGCCCTCAAGAGGTCACTGTGCAAAGAAAGTCATTCCTGGTTGGTGTGGTGTTCGGACCATAAATCTTGG',\n",
" 'sentence2': 'MEENQTMVTEFVLLGFCLGPRIHLVLFLLFSLFYTLTILGNGTILAMICLDSRLHTPMYFFLSHLAIVDMAYACNTVPQTLINLLDETRPITFAGCMTQTFLFLAFAHTECVLLVVMSYDRYVAICHPLHYTVIMNWRVCTILAAVSWIFSFLLALVHLVLILRLPFCGPHEINHFFCEILSVLKLACADTTLNQVVIFAACVFILVAPLCFVLVSYTRILVAILRIQSGEGRRKAFSTCSSHLCVVGLFFGSAIVMYMAPKSQHPEEQQKVLFLFYSFFNPMLNPLIYSLRNAEVKGALKRSLCKESHSWLVWCSDHKSW',\n",
" 'label': 1}"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ft_dataset = load_dataset('json', data_files='data/dna_protein_my.json')\n",
"ft_dataset[\"train\"][0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8f3ec639-e426-4233-a20a-dad94069175b",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
01-data_env/3-dataset-use.ipynb
+261
View File
@@ -0,0 +1,261 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "68c06a52-e27c-4da6-8a02-cd010270bedf",
"metadata": {},
"source": [
"# 3 datasets库基本使用"
]
},
{
"cell_type": "markdown",
"id": "2dc4c70f-694c-4785-81d8-26ebab2b7210",
"metadata": {},
"source": [
"## 基本使用\n",
"上一节中，已经介绍了使用datasets读取本地文件的方法，这一节继续介绍datasets一些常用的方法\n",
"\n",
"首先是数据分割，因为我们从数据源获得DNA序列等数据，都是一个文本文件，但训练的时候，一般都需要分成训练集和测试集等\n",
"\n",
"一个简单的例子如下所示："
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "6e9f346f-31f6-40cc-86e5-723c65033883",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['text'],\n",
" num_rows: 1025615\n",
" })\n",
" test: Dataset({\n",
" features: ['text'],\n",
" num_rows: 53980\n",
" })\n",
"})"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#读取dna数据\n",
"from datasets import load_dataset\n",
"dna_dataset = load_dataset('text', data_files='data/dna_1g.txt')['train'].rain_test_split(test_size=0.05) #默认已经shuffle\n",
"dna_dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "75900650-74da-4ca9-a285-b2832a5a1485",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'text': 'ATGTGTGCAATGGGTTATCTTTATGTAATAACAGTCATATCACGGGTGTTCCTCAGAAGTAGTGAACTGGCTAGCATTTTTAGACACTATGTGATCTCTCATATGACTACACTCAATTTAAAATAAAATGAAATGTGTTGTGTGTGTCTAAAATCTATAAAGGGAAAAGTATCTTAAGTATTTTTTAGATGTTAAAGTAGATGTGTATCCTAAAATATGCATTGTTCACAGATGTTAAAATTACAACTACAATCTGTGAAACACAGATCTTAGGACAGCAATGTTTCACAAGAAAAAAAATGATGCAGCCTTCTTTAGTATTTATAGTCATTTGAACAATTATGGCAACCATAAGTTCATATATAACATCCCCATTTGGTGAAACTAGTTGGGAAAGATTAGAAGGTATGACCTTGTTGGAGGAACTATACCATTGGGGTGGCTTTGAGACTTCAGAAGTTTCAAGGCCCATTTAGTGCTTTCTACCTTATGAAGCTGTGAGTTCTCCTTGCTAGCTACATAACTTGGAAAGCAGGCCCTGCACTTCACCCAAGGAGCACATTAGAGCTGGCCCTTTTGGAAGGCAATTGCGTAAGCCACACCAGGGCACCAGAGATCTGGCACTGCCATGCTCCTGCTTGCAAGTAGTGGTGTGGGTGTTGGGTGATGCCCTCCAGTCCCACCTTTTGCCACCTGTAGTAGTCAGGGGAGTTGGCCTAAGGGCATGAGAGCCTAAGACTTCACCCTAATCCCTCACCAACTGTAGCATGTGGAAGAGCAGGCTCTGTACCTTCCCTGGGCAACACATTGGAGCTGGCCCCTCACAGGCTGCAGGACTTGGGAGAGTGAGTGCTGCACCTTGACTGTGAAGGTGGTTTTGGAGGTGTGGGTGTGAGACCATGAGACCAAGAGAGGAATGGAATATTACTCACTTATTAAAAACAATGACTTCATGAAATTTGCAGGCAAATGGATGGAACTTGAAAATATCCTGAGTGAG'}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dna_dataset[\"test\"][0]"
]
},
{
"cell_type": "markdown",
"id": "cdcc5404-6590-47a4-be2c-2c1d35d3bae4",
"metadata": {},
"source": [
"可以看到，数据集已经分割成了train和test两个数据集，而在分割的时候，已经进行的随机处理\n",
"\n",
"当然，如果数据集过大，我们只需要其中一部分，这个也是一个常见的需求，一般可以使用 Dataset.select() 函数"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "049ad194-cb60-4b0f-8554-1915bfc7a9cd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['text'],\n",
" num_rows: 50000\n",
" })\n",
" valid: Dataset({\n",
" features: ['text'],\n",
" num_rows: 500\n",
" })\n",
"})"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset, DatasetDict\n",
"\n",
"dna_dataset_sample = DatasetDict(\n",
" {\n",
" \"train\": dna_dataset[\"train\"].shuffle().select(range(50000)), \n",
" \"valid\": dna_dataset[\"test\"].shuffle().select(range(500)),\n",
" \"evla\": dna_dataset[\"test\"].shuffle().select(range(500))\n",
"\n",
" }\n",
")\n",
"dna_dataset_sample"
]
},
{
"cell_type": "markdown",
"id": "50cceda3-36ca-4fa6-bfb5-1dbeb155fe4f",
"metadata": {},
"source": [
"可以看到，我们使用DatasetDict来直接构造datasets，先使用shuffle()来随机，然后使用select来选择前n个数据\n",
"\n",
"select的参数为indices (list 或 range): 索引列表或范围对象，指明要选择哪些样本，如dataset.select([0, 2, 4])就是选择1,3,5条记录"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "17a1fa7c-ff4b-419f-8a82-e58cc5777cd4",
"metadata": {},
"source": [
"## 读取线上库\n",
"\n",
"当然，数据也可以直接从huggingface的线上仓库读取，这时候需要注意科学上网问题。\n",
"\n",
"具体使用函数也是load_dataset\n",
"\n",
"<img src='img/datasets_dnagpt.png' width='800px' />"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "6ae24950-2c74-457b-b1f2-d2e4397e1fa1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"#或者\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "30ff9798-d06d-4992-81fc-03102f03599b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 59196\n",
" })\n",
"})"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset\n",
"dna_data = load_dataset(\"dnagpt/dna_core_promoter\")\n",
"dna_data"
]
},
{
"cell_type": "markdown",
"id": "30c4b754-af11-4ac1-9742-45427059617e",
"metadata": {},
"source": [
"当然，如果你想分享你的数据集到huggingface上面，也是一行函数即可："
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f9847be9-e085-41e3-ad29-a450cc017d64",
"metadata": {},
"outputs": [],
"source": [
"dna_data.push_to_hub(\"org_name/your_dataset_name\", token=\"hf_yourtoken\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
01-data_env/data/dna_1g.txt
+100000
View File
File diff suppressed because it is too large Load Diff
01-data_env/data/dna_protein_my.json
+4000
View File
File diff suppressed because it is too large Load Diff
01-data_env/data/english_500m.txt
+117039
View File
File diff suppressed because it is too large Load Diff
01-data_env/data/protein_1g.txt
+100000
View File
File diff suppressed because it is too large Load Diff
01-data_env/img/.ipynb_checkpoints/zhushi-checkpoint.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 194 KiB

01-data_env/img/datasets_dnagpt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 348 KiB

01-data_env/img/zhushi.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 194 KiB

02-gpt2_bert/.ipynb_checkpoints/1-dna-bpe-checkpoint.ipynb
+528
View File
@@ -0,0 +1,528 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "a9fffce5-83e3-4838-8335-acb2e3b50c35",
"metadata": {},
"source": [
"# 2.1 DNA分词器构建"
]
},
{
"cell_type": "markdown",
"id": "f28b0950-37dc-4f78-ae6c-9fca33d513fc",
"metadata": {},
"source": [
"## **分词算法**\n",
"\n",
"### **什么是分词**\n",
"分词就是把一个文本序列，分成一个一个的token/词，对于英文这种天生带空格的语言，一般使用空格和标点分词就行了，而对于中文等语言，并没有特殊的符号来分词，因此，一般需要设计专门的分词算法，对于大模型而言，一般需要处理多种语言，因此，也需要专门的分词算法。\n",
"\n",
"在大模型（如 BERT、GPT 系列、T5 等）中，分词器（tokenizer）扮演着至关重要的角色。它负责将原始文本转换为模型可以处理的格式，即将文本分解成 token 序列，并将这些 token 映射到模型词汇表中的唯一 ID。分词器的选择和配置直接影响模型的性能和效果。以下是几种常见的分词器及其特点，特别关注它们在大型语言模型中的应用。\n",
"\n",
"### 1. **WordPiece 分词器**\n",
"\n",
"- **使用场景**：广泛应用于 BERT 及其变体。\n",
"- **工作原理**：基于频率统计，从语料库中学习最有效的词汇表。它根据子词（subword）在文本中的出现频率来决定如何分割单词。例如，“playing” 可能被分为 “play” 和 “##ing”，其中“##”表示该部分是前一个 token 的延续。\n",
"- **优点**：\n",
" - 处理未知词汇能力强，能够将未见过的词汇分解为已知的子词。\n",
" - 兼容性好，适合多种语言任务。\n",
"- **缺点**：\n",
" - 需要额外的标记（如 `##`）来指示子词，可能影响某些应用场景下的可读性。\n",
"\n",
"### 2. **Byte Pair Encoding (BPE)**\n",
"\n",
"- **使用场景**：广泛应用于 GPT 系列、RoBERTa、XLM-R 等模型。\n",
"- **工作原理**：通过迭代地合并最常见的字符对来构建词汇表。BPE 是一种无监督的学习方法，能够在不依赖于预先定义的词汇表的情况下进行分词。\n",
"- **优点**：\n",
" - 灵活性高，适应性强，尤其适用于多语言模型。\n",
" - 不需要特殊标记，生成的词汇表更简洁。\n",
"- **缺点**：\n",
" - 对于某些语言或领域特定的词汇，可能会产生较短的子词，导致信息丢失。\n",
"\n",
"### 3. **SentencePiece**\n",
"\n",
"- **使用场景**：常见于 T5、mBART 等多语言模型。\n",
"- **工作原理**：结合了 BPE 和 WordPiece 的优点，同时支持字符级和词汇级分词。它可以在没有空格的语言（如中文、日文）中表现良好。\n",
"- **优点**：\n",
" - 支持无空格语言，适合多语言处理。\n",
" - 学习速度快，适应性强。\n",
"- **缺点**：\n",
" - 对于某些特定领域的专业术语，可能需要额外的预处理步骤。\n",
"\n",
"### 4. **Character-Level Tokenizer**\n",
"\n",
"- **使用场景**：较少用于大型语言模型，但在某些特定任务（如拼写检查、手写识别）中有应用。\n",
"- **工作原理**：直接将每个字符视为一个 token。这种方式简单直接，但通常会导致较大的词汇表。\n",
"- **优点**：\n",
" - 简单易实现，不需要复杂的训练过程。\n",
" - 对于字符级别的任务非常有效。\n",
"- **缺点**：\n",
" - 词汇表较大，计算资源消耗较多。\n",
" - 捕捉上下文信息的能力较弱。\n",
"\n",
"### 5. **Unigram Language Model**\n",
"\n",
"- **使用场景**：主要用于 SentencePiece 中。\n",
"- **工作原理**：基于概率分布，选择最优的分词方案以最大化似然函数。这种方法类似于 BPE，但在构建词汇表时考虑了更多的统计信息。\n",
"- **优点**：\n",
" - 统计基础强，优化效果好。\n",
" - 适应性强，适用于多种语言和任务。\n",
"- **缺点**：\n",
" - 计算复杂度较高，训练时间较长。\n",
"\n",
"### 分词器的关键特性\n",
"\n",
"无论选择哪种分词器，以下几个关键特性都是设计和应用中需要考虑的：\n",
"\n",
"- **词汇表大小**：决定了模型所能识别的词汇量。较大的词汇表可以捕捉更多细节，但也增加了内存和计算需求。\n",
"- **处理未知词汇的能力**：好的分词器应该能够有效地处理未登录词（OOV, Out-Of-Vocabulary），将其分解为已知的子词。\n",
"- **多语言支持**：对于多语言模型，分词器应能处理不同语言的文本，尤其是那些没有明显分隔符的语言。\n",
"- **效率和速度**：分词器的执行速度直接影响整个数据处理管道的效率，尤其是在大规模数据集上。\n",
"- **兼容性和灵活性**：分词器应与目标模型架构兼容，并且能够灵活适应不同的任务需求。"
]
},
{
"cell_type": "markdown",
"id": "165e2594-277d-44d0-b582-77859a0bc0b2",
"metadata": {},
"source": [
"## DNA等生物序列分词\n",
"在生物信息学中，DNA 和蛋白质序列的处理与自然语言处理（NLP）有相似之处，但也有其独特性。为了提取这些生物序列的特征并用于机器学习或深度学习模型，通常需要将长序列分解成更小的片段（类似于 NLP 中的“分词”），以便更好地捕捉局部和全局特征。以下是几种常见的方法，用于对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。\n",
"\n",
"### 1. **K-mer 分解**\n",
"\n",
"**定义**：K-mer 是指长度为 k 的连续子序列。例如，在 DNA 序列中，一个 3-mer 可能是 \"ATG\" 或 \"CGA\"。\n",
"\n",
"**应用**：\n",
"- **DNA 序列**：常用的 k 值范围从 3 到 6。较小的 k 值可以捕捉到更细粒度的信息，而较大的 k 值则有助于识别更长的模式。\n",
"- **蛋白质序列**：k 值通常较大，因为氨基酸的数量较多（20 种），较长的 k-mer 可以捕捉到重要的结构域或功能区域。\n",
"\n",
"**优点**：\n",
"- 简单且直观，易于实现。\n",
"- 可以捕捉到短序列中的局部特征。\n",
"\n",
"**缺点**：\n",
"- 对于非常长的序列，生成的 k-mer 数量会非常大，导致维度爆炸问题。\n",
"- 不同位置的 k-mer 之间缺乏上下文关系。"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "29c390ef-2e9d-493e-9991-69ecb835b52b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"DNA 3-mers: ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
"Protein 4-mers: ['MKQH', 'KQHK', 'QHKA', 'HKAM', 'KAMI', 'AMIV', 'MIVA', 'IVAL', 'VALI', 'ALIV', 'LIVL', 'IVLI', 'VLIT', 'LITA', 'ITAY']\n"
]
}
],
"source": [
"#示例代码（Python）\n",
"\n",
"def k_mer(seq, k):\n",
" return [seq[i:i+k] for i in range(len(seq) - k + 1)]\n",
"\n",
"dna_sequence = \"ATGCGTACGTA\"\n",
"protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
"\n",
"print(\"DNA 3-mers:\", k_mer(dna_sequence, 3))\n",
"print(\"Protein 4-mers:\", k_mer(protein_sequence, 4))"
]
},
{
"cell_type": "markdown",
"id": "7ced2bfb-bd42-425a-a3ad-54c9573609c5",
"metadata": {},
"source": [
"### 2. **滑动窗口**\n",
"\n",
"**定义**：滑动窗口方法通过设定一个固定大小的窗口沿着序列移动，并在每个位置提取窗口内的子序列。这与 K-mer 类似，但允许重叠。\n",
"\n",
"**应用**：\n",
"- **DNA 和蛋白质序列**：窗口大小可以根据具体任务调整，如基因预测、蛋白质结构预测等。\n",
"\n",
"**优点**：\n",
"- 提供了更多的灵活性，可以控制窗口的步长和大小。\n",
"- 有助于捕捉局部和全局特征。\n",
"\n",
"**缺点**：\n",
"- 计算复杂度较高，尤其是当窗口大小较大时。"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "82cecf91-0076-4c12-b11c-b35120581ef9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sliding window (DNA, size=3, step=1): ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
"Sliding window (Protein, size=4, step=2): ['MKQH', 'QHKA', 'KAMI', 'MIVA', 'VALI', 'LIVL', 'VLIT', 'ITAY']\n"
]
}
],
"source": [
"def sliding_window(seq, window_size, step=1):\n",
" return [seq[i:i+window_size] for i in range(0, len(seq) - window_size + 1, step)]\n",
"\n",
"dna_sequence = \"ATGCGTACGTA\"\n",
"protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
"\n",
"print(\"Sliding window (DNA, size=3, step=1):\", sliding_window(dna_sequence, 3))\n",
"print(\"Sliding window (Protein, size=4, step=2):\", sliding_window(protein_sequence, 4, step=2))"
]
},
{
"cell_type": "markdown",
"id": "c33ab920-b451-4846-93d4-20da5a4e1001",
"metadata": {},
"source": [
"### 3. **词表分词和嵌入式表示**\n",
"\n",
"**定义**：使用预训练的嵌入模型（如 Word2Vec、BERT 等）来将每个 token 映射到高维向量空间中。对于生物序列，可以使用专门设计的嵌入模型，如 ProtTrans、ESM 等。\n",
"\n",
"**应用**：\n",
"- **DNA 和蛋白质序列**：嵌入模型可以捕捉到序列中的语义信息和上下文依赖关系。\n",
"\n",
"**优点**：\n",
"- 捕捉到丰富的语义信息，适合复杂的下游任务。\n",
"- 可以利用大规模预训练模型的优势。\n",
"\n",
"**缺点**：\n",
"- 需要大量的计算资源来进行预训练。\n",
"- 模型复杂度较高，解释性较差。"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "02bf2af0-6077-4b27-8822-f1c3f22914fa",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "d43b60ee-67f2-4d06-95ea-966c01084fc4",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
"To disable this warning, you can either:\n",
"\t- Avoid using `tokenizers` before the fork if possible\n",
"\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"['ATGCG', 'TACG', 'T', 'A']\n",
"Embeddings shape: torch.Size([1, 4, 768])\n"
]
}
],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"import torch\n",
"\n",
"# 加载预训练的蛋白质嵌入模型\n",
"tokenizer = AutoTokenizer.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
"model = AutoModel.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
"\n",
"dna_sequence = \"ATGCGTACGTA\"\n",
"print(tokenizer.tokenize(dna_sequence))\n",
"\n",
"# 编码序列\n",
"inputs = tokenizer(dna_sequence, return_tensors=\"pt\")\n",
"\n",
"# 获取嵌入\n",
"with torch.no_grad():\n",
" outputs = model(**inputs)\n",
" embeddings = outputs.last_hidden_state\n",
"\n",
"print(\"Embeddings shape:\", embeddings.shape)"
]
},
{
"cell_type": "markdown",
"id": "c24f10dc-1117-4493-9333-5ed6d898f44a",
"metadata": {},
"source": [
"### **训练DNA BPE分词器**\n",
"\n",
"以上方法展示了如何对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。选择哪种方法取决于具体的任务需求和数据特性。对于简单的分类或回归任务，K-mer 分解或滑动窗口可能是足够的；而对于更复杂的任务，如序列标注或结构预测，基于词汇表的方法或嵌入表示可能会提供更好的性能。\n",
"\n",
"目前大部分生物序列大模型的论文中，使用最多的依然是传统的K-mer，但一些SOTA的论文则以BEP为主。而BEP分词也是目前GPT、llama等主流自然语言大模型使用的基础分词器。\n",
"\n",
"因此，我们也演示下从头训练一个DNA BPE分词器的方法。\n",
"\n",
"我们首先看下GPT2模型，默认的分词器，对DNA序列分词的结果："
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "43f1eb8b-1cc2-4ab5-aa8e-2a63132be98c",
"metadata": {},
"outputs": [],
"source": [
"from tokenizers import (\n",
" decoders,\n",
" models,\n",
" normalizers,\n",
" pre_tokenizers,\n",
" processors,\n",
" trainers,\n",
" Tokenizer,\n",
")\n",
"from transformers import AutoTokenizer"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "27e88f7b-1399-418b-9b91-f970762fac0c",
"metadata": {},
"outputs": [],
"source": [
"gpt2_tokenizer = AutoTokenizer.from_pretrained('gpt2')\n",
"gpt2_tokenizer.pad_token = gpt2_tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "4b015db7-63ba-4909-b02f-07634b3d5584",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['T', 'GG', 'C', 'GT', 'GA', 'AC', 'CC', 'GG', 'G', 'AT', 'C', 'GG', 'G']"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gpt2_tokenizer.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
]
},
{
"cell_type": "markdown",
"id": "a246fbc9-9e29-4b63-bdf7-f80635d06d1e",
"metadata": {},
"source": [
"可以看到，gpt2模型因为是以英文为主的BPE分词模型，分解的都是1到2个字母的结果，这样显然很难充分表达生物语义，因此，我们使用DNA序列来训练1个BPE分词器，代码也非常简单："
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8357a695-1c29-4b5c-8099-d2e337189410",
"metadata": {},
"outputs": [],
"source": [
"tokenizer = Tokenizer(models.BPE())\n",
"tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串\n",
"trainer = trainers.BpeTrainer(vocab_size=30000, special_tokens=[\"<|endoftext|>\"]) #3w words"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "32c95888-1498-45cf-8453-421219cc7d45",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\n"
]
}
],
"source": [
"tokenizer.train([\"../01-data_env/data/dna_1g.txt\"], trainer=trainer) #all file list, take 10-20 min"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "5ffdd717-72ed-4a37-bafc-b4a0f61f8ff1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']\n"
]
}
],
"source": [
"encoding = tokenizer.encode(\"TGGCGTGAACCCGGGATCGGG\")\n",
"print(encoding.tokens)"
]
},
{
"cell_type": "markdown",
"id": "a96e7838-6c23-4446-bf86-b098cd93214a",
"metadata": {},
"source": [
"可以看到，以DNA数据训练的分词器，分词效果明显要好的多，各种长度的词都有。"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f1d757c1-702b-4147-9207-471f422f67b2",
"metadata": {},
"outputs": [],
"source": [
"tokenizer.save(\"dna_bpe_dict.json\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "caf8ecea-359e-487b-b456-fab546b9da0d",
"metadata": {},
"outputs": [],
"source": [
"#然后我们可以使用from_file() 方法从该文件里重新加载 Tokenizer 对象：\n",
"new_tokenizer = Tokenizer.from_file(\"dna_bpe_dict.json\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "8ec6f045-bc30-4012-8027-a879df8def3a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('dna_bpe_dict/tokenizer_config.json',\n",
" 'dna_bpe_dict/special_tokens_map.json',\n",
" 'dna_bpe_dict/vocab.json',\n",
" 'dna_bpe_dict/merges.txt',\n",
" 'dna_bpe_dict/added_tokens.json',\n",
" 'dna_bpe_dict/tokenizer.json')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#要在 🤗 Transformers 中使用这个标记器，我们必须将它包裹在一个 PreTrainedTokenizerFast 类中\n",
"from transformers import GPT2TokenizerFast\n",
"dna_tokenizer = GPT2TokenizerFast(tokenizer_object=new_tokenizer)\n",
"dna_tokenizer.save_pretrained(\"dna_bpe_dict\")\n",
"#dna_tokenizer.push_to_hub(\"dna_bpe_dict_1g\", organization=\"dnagpt\", use_auth_token=\"hf_*****\") # push to huggingface"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "f84506d8-6208-4027-aad7-2b68a1bc16d6",
"metadata": {},
"outputs": [],
"source": [
"tokenizer_new = AutoTokenizer.from_pretrained('dna_bpe_dict')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "d40d4d53-6fed-445c-afb5-c0346ab854c8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer_new.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "640302f6-f740-41a4-ae92-ca4c43d97493",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02-gpt2_bert/.ipynb_checkpoints/2-dna-gpt-checkpoint.ipynb
+5167
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/.ipynb_checkpoints/3-dna-bert-checkpoint.ipynb
+754
View File
File diff suppressed because one or more lines are too long
02-gpt2_bert/.ipynb_checkpoints/5-multi-seq-gpt-checkpoint.ipynb
+13106
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/1-dna-bpe.ipynb
+528
View File
@@ -0,0 +1,528 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "a9fffce5-83e3-4838-8335-acb2e3b50c35",
"metadata": {},
"source": [
"# 2.1 DNA分词器构建"
]
},
{
"cell_type": "markdown",
"id": "f28b0950-37dc-4f78-ae6c-9fca33d513fc",
"metadata": {},
"source": [
"## **分词算法**\n",
"\n",
"### **什么是分词**\n",
"分词就是把一个文本序列，分成一个一个的token/词，对于英文这种天生带空格的语言，一般使用空格和标点分词就行了，而对于中文等语言，并没有特殊的符号来分词，因此，一般需要设计专门的分词算法，对于大模型而言，一般需要处理多种语言，因此，也需要专门的分词算法。\n",
"\n",
"在大模型（如 BERT、GPT 系列、T5 等）中，分词器（tokenizer）扮演着至关重要的角色。它负责将原始文本转换为模型可以处理的格式，即将文本分解成 token 序列，并将这些 token 映射到模型词汇表中的唯一 ID。分词器的选择和配置直接影响模型的性能和效果。以下是几种常见的分词器及其特点，特别关注它们在大型语言模型中的应用。\n",
"\n",
"### 1. **WordPiece 分词器**\n",
"\n",
"- **使用场景**：广泛应用于 BERT 及其变体。\n",
"- **工作原理**：基于频率统计，从语料库中学习最有效的词汇表。它根据子词（subword）在文本中的出现频率来决定如何分割单词。例如，“playing” 可能被分为 “play” 和 “##ing”，其中“##”表示该部分是前一个 token 的延续。\n",
"- **优点**：\n",
" - 处理未知词汇能力强，能够将未见过的词汇分解为已知的子词。\n",
" - 兼容性好，适合多种语言任务。\n",
"- **缺点**：\n",
" - 需要额外的标记（如 `##`）来指示子词，可能影响某些应用场景下的可读性。\n",
"\n",
"### 2. **Byte Pair Encoding (BPE)**\n",
"\n",
"- **使用场景**：广泛应用于 GPT 系列、RoBERTa、XLM-R 等模型。\n",
"- **工作原理**：通过迭代地合并最常见的字符对来构建词汇表。BPE 是一种无监督的学习方法，能够在不依赖于预先定义的词汇表的情况下进行分词。\n",
"- **优点**：\n",
" - 灵活性高，适应性强，尤其适用于多语言模型。\n",
" - 不需要特殊标记，生成的词汇表更简洁。\n",
"- **缺点**：\n",
" - 对于某些语言或领域特定的词汇，可能会产生较短的子词，导致信息丢失。\n",
"\n",
"### 3. **SentencePiece**\n",
"\n",
"- **使用场景**：常见于 T5、mBART 等多语言模型。\n",
"- **工作原理**：结合了 BPE 和 WordPiece 的优点，同时支持字符级和词汇级分词。它可以在没有空格的语言（如中文、日文）中表现良好。\n",
"- **优点**：\n",
" - 支持无空格语言，适合多语言处理。\n",
" - 学习速度快，适应性强。\n",
"- **缺点**：\n",
" - 对于某些特定领域的专业术语，可能需要额外的预处理步骤。\n",
"\n",
"### 4. **Character-Level Tokenizer**\n",
"\n",
"- **使用场景**：较少用于大型语言模型，但在某些特定任务（如拼写检查、手写识别）中有应用。\n",
"- **工作原理**：直接将每个字符视为一个 token。这种方式简单直接，但通常会导致较大的词汇表。\n",
"- **优点**：\n",
" - 简单易实现，不需要复杂的训练过程。\n",
" - 对于字符级别的任务非常有效。\n",
"- **缺点**：\n",
" - 词汇表较大，计算资源消耗较多。\n",
" - 捕捉上下文信息的能力较弱。\n",
"\n",
"### 5. **Unigram Language Model**\n",
"\n",
"- **使用场景**：主要用于 SentencePiece 中。\n",
"- **工作原理**：基于概率分布，选择最优的分词方案以最大化似然函数。这种方法类似于 BPE，但在构建词汇表时考虑了更多的统计信息。\n",
"- **优点**：\n",
" - 统计基础强，优化效果好。\n",
" - 适应性强，适用于多种语言和任务。\n",
"- **缺点**：\n",
" - 计算复杂度较高，训练时间较长。\n",
"\n",
"### 分词器的关键特性\n",
"\n",
"无论选择哪种分词器，以下几个关键特性都是设计和应用中需要考虑的：\n",
"\n",
"- **词汇表大小**：决定了模型所能识别的词汇量。较大的词汇表可以捕捉更多细节，但也增加了内存和计算需求。\n",
"- **处理未知词汇的能力**：好的分词器应该能够有效地处理未登录词（OOV, Out-Of-Vocabulary），将其分解为已知的子词。\n",
"- **多语言支持**：对于多语言模型，分词器应能处理不同语言的文本，尤其是那些没有明显分隔符的语言。\n",
"- **效率和速度**：分词器的执行速度直接影响整个数据处理管道的效率，尤其是在大规模数据集上。\n",
"- **兼容性和灵活性**：分词器应与目标模型架构兼容，并且能够灵活适应不同的任务需求。"
]
},
{
"cell_type": "markdown",
"id": "165e2594-277d-44d0-b582-77859a0bc0b2",
"metadata": {},
"source": [
"## DNA等生物序列分词\n",
"在生物信息学中，DNA 和蛋白质序列的处理与自然语言处理（NLP）有相似之处，但也有其独特性。为了提取这些生物序列的特征并用于机器学习或深度学习模型，通常需要将长序列分解成更小的片段（类似于 NLP 中的“分词”），以便更好地捕捉局部和全局特征。以下是几种常见的方法，用于对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。\n",
"\n",
"### 1. **K-mer 分解**\n",
"\n",
"**定义**：K-mer 是指长度为 k 的连续子序列。例如，在 DNA 序列中，一个 3-mer 可能是 \"ATG\" 或 \"CGA\"。\n",
"\n",
"**应用**：\n",
"- **DNA 序列**：常用的 k 值范围从 3 到 6。较小的 k 值可以捕捉到更细粒度的信息，而较大的 k 值则有助于识别更长的模式。\n",
"- **蛋白质序列**：k 值通常较大，因为氨基酸的数量较多（20 种），较长的 k-mer 可以捕捉到重要的结构域或功能区域。\n",
"\n",
"**优点**：\n",
"- 简单且直观，易于实现。\n",
"- 可以捕捉到短序列中的局部特征。\n",
"\n",
"**缺点**：\n",
"- 对于非常长的序列，生成的 k-mer 数量会非常大，导致维度爆炸问题。\n",
"- 不同位置的 k-mer 之间缺乏上下文关系。"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "29c390ef-2e9d-493e-9991-69ecb835b52b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"DNA 3-mers: ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
"Protein 4-mers: ['MKQH', 'KQHK', 'QHKA', 'HKAM', 'KAMI', 'AMIV', 'MIVA', 'IVAL', 'VALI', 'ALIV', 'LIVL', 'IVLI', 'VLIT', 'LITA', 'ITAY']\n"
]
}
],
"source": [
"#示例代码（Python）\n",
"\n",
"def k_mer(seq, k):\n",
" return [seq[i:i+k] for i in range(len(seq) - k + 1)]\n",
"\n",
"dna_sequence = \"ATGCGTACGTA\"\n",
"protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
"\n",
"print(\"DNA 3-mers:\", k_mer(dna_sequence, 3))\n",
"print(\"Protein 4-mers:\", k_mer(protein_sequence, 4))"
]
},
{
"cell_type": "markdown",
"id": "7ced2bfb-bd42-425a-a3ad-54c9573609c5",
"metadata": {},
"source": [
"### 2. **滑动窗口**\n",
"\n",
"**定义**：滑动窗口方法通过设定一个固定大小的窗口沿着序列移动，并在每个位置提取窗口内的子序列。这与 K-mer 类似，但允许重叠。\n",
"\n",
"**应用**：\n",
"- **DNA 和蛋白质序列**：窗口大小可以根据具体任务调整，如基因预测、蛋白质结构预测等。\n",
"\n",
"**优点**：\n",
"- 提供了更多的灵活性，可以控制窗口的步长和大小。\n",
"- 有助于捕捉局部和全局特征。\n",
"\n",
"**缺点**：\n",
"- 计算复杂度较高，尤其是当窗口大小较大时。"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "82cecf91-0076-4c12-b11c-b35120581ef9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sliding window (DNA, size=3, step=1): ['ATG', 'TGC', 'GCG', 'CGT', 'GTA', 'TAC', 'ACG', 'CGT', 'GTA']\n",
"Sliding window (Protein, size=4, step=2): ['MKQH', 'QHKA', 'KAMI', 'MIVA', 'VALI', 'LIVL', 'VLIT', 'ITAY']\n"
]
}
],
"source": [
"def sliding_window(seq, window_size, step=1):\n",
" return [seq[i:i+window_size] for i in range(0, len(seq) - window_size + 1, step)]\n",
"\n",
"dna_sequence = \"ATGCGTACGTA\"\n",
"protein_sequence = \"MKQHKAMIVALIVLITAY\"\n",
"\n",
"print(\"Sliding window (DNA, size=3, step=1):\", sliding_window(dna_sequence, 3))\n",
"print(\"Sliding window (Protein, size=4, step=2):\", sliding_window(protein_sequence, 4, step=2))"
]
},
{
"cell_type": "markdown",
"id": "c33ab920-b451-4846-93d4-20da5a4e1001",
"metadata": {},
"source": [
"### 3. **词表分词和嵌入式表示**\n",
"\n",
"**定义**：使用预训练的嵌入模型（如 Word2Vec、BERT 等）来将每个 token 映射到高维向量空间中。对于生物序列，可以使用专门设计的嵌入模型，如 ProtTrans、ESM 等。\n",
"\n",
"**应用**：\n",
"- **DNA 和蛋白质序列**：嵌入模型可以捕捉到序列中的语义信息和上下文依赖关系。\n",
"\n",
"**优点**：\n",
"- 捕捉到丰富的语义信息，适合复杂的下游任务。\n",
"- 可以利用大规模预训练模型的优势。\n",
"\n",
"**缺点**：\n",
"- 需要大量的计算资源来进行预训练。\n",
"- 模型复杂度较高，解释性较差。"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "02bf2af0-6077-4b27-8822-f1c3f22914fa",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "d43b60ee-67f2-4d06-95ea-966c01084fc4",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
"To disable this warning, you can either:\n",
"\t- Avoid using `tokenizers` before the fork if possible\n",
"\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"['ATGCG', 'TACG', 'T', 'A']\n",
"Embeddings shape: torch.Size([1, 4, 768])\n"
]
}
],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"import torch\n",
"\n",
"# 加载预训练的蛋白质嵌入模型\n",
"tokenizer = AutoTokenizer.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
"model = AutoModel.from_pretrained(\"dnagpt/gpt_dna_v0\")\n",
"\n",
"dna_sequence = \"ATGCGTACGTA\"\n",
"print(tokenizer.tokenize(dna_sequence))\n",
"\n",
"# 编码序列\n",
"inputs = tokenizer(dna_sequence, return_tensors=\"pt\")\n",
"\n",
"# 获取嵌入\n",
"with torch.no_grad():\n",
" outputs = model(**inputs)\n",
" embeddings = outputs.last_hidden_state\n",
"\n",
"print(\"Embeddings shape:\", embeddings.shape)"
]
},
{
"cell_type": "markdown",
"id": "c24f10dc-1117-4493-9333-5ed6d898f44a",
"metadata": {},
"source": [
"### **训练DNA BPE分词器**\n",
"\n",
"以上方法展示了如何对 DNA 和蛋白质序列进行“分词”，以提取有用的特征。选择哪种方法取决于具体的任务需求和数据特性。对于简单的分类或回归任务，K-mer 分解或滑动窗口可能是足够的；而对于更复杂的任务，如序列标注或结构预测，基于词汇表的方法或嵌入表示可能会提供更好的性能。\n",
"\n",
"目前大部分生物序列大模型的论文中，使用最多的依然是传统的K-mer，但一些SOTA的论文则以BEP为主。而BEP分词也是目前GPT、llama等主流自然语言大模型使用的基础分词器。\n",
"\n",
"因此，我们也演示下从头训练一个DNA BPE分词器的方法。\n",
"\n",
"我们首先看下GPT2模型，默认的分词器，对DNA序列分词的结果："
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "43f1eb8b-1cc2-4ab5-aa8e-2a63132be98c",
"metadata": {},
"outputs": [],
"source": [
"from tokenizers import (\n",
" decoders,\n",
" models,\n",
" normalizers,\n",
" pre_tokenizers,\n",
" processors,\n",
" trainers,\n",
" Tokenizer,\n",
")\n",
"from transformers import AutoTokenizer"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "27e88f7b-1399-418b-9b91-f970762fac0c",
"metadata": {},
"outputs": [],
"source": [
"gpt2_tokenizer = AutoTokenizer.from_pretrained('gpt2')\n",
"gpt2_tokenizer.pad_token = gpt2_tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "4b015db7-63ba-4909-b02f-07634b3d5584",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['T', 'GG', 'C', 'GT', 'GA', 'AC', 'CC', 'GG', 'G', 'AT', 'C', 'GG', 'G']"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gpt2_tokenizer.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
]
},
{
"cell_type": "markdown",
"id": "a246fbc9-9e29-4b63-bdf7-f80635d06d1e",
"metadata": {},
"source": [
"可以看到，gpt2模型因为是以英文为主的BPE分词模型，分解的都是1到2个字母的结果，这样显然很难充分表达生物语义，因此，我们使用DNA序列来训练1个BPE分词器，代码也非常简单："
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8357a695-1c29-4b5c-8099-d2e337189410",
"metadata": {},
"outputs": [],
"source": [
"tokenizer = Tokenizer(models.BPE())\n",
"tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串\n",
"trainer = trainers.BpeTrainer(vocab_size=30000, special_tokens=[\"<|endoftext|>\"]) #3w words"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "32c95888-1498-45cf-8453-421219cc7d45",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\n"
]
}
],
"source": [
"tokenizer.train([\"../01-data_env/data/dna_1g.txt\"], trainer=trainer) #all file list, take 10-20 min"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "5ffdd717-72ed-4a37-bafc-b4a0f61f8ff1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']\n"
]
}
],
"source": [
"encoding = tokenizer.encode(\"TGGCGTGAACCCGGGATCGGG\")\n",
"print(encoding.tokens)"
]
},
{
"cell_type": "markdown",
"id": "a96e7838-6c23-4446-bf86-b098cd93214a",
"metadata": {},
"source": [
"可以看到，以DNA数据训练的分词器，分词效果明显要好的多，各种长度的词都有。"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f1d757c1-702b-4147-9207-471f422f67b2",
"metadata": {},
"outputs": [],
"source": [
"tokenizer.save(\"dna_bpe_dict.json\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "caf8ecea-359e-487b-b456-fab546b9da0d",
"metadata": {},
"outputs": [],
"source": [
"#然后我们可以使用from_file() 方法从该文件里重新加载 Tokenizer 对象：\n",
"new_tokenizer = Tokenizer.from_file(\"dna_bpe_dict.json\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "8ec6f045-bc30-4012-8027-a879df8def3a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('dna_bpe_dict/tokenizer_config.json',\n",
" 'dna_bpe_dict/special_tokens_map.json',\n",
" 'dna_bpe_dict/vocab.json',\n",
" 'dna_bpe_dict/merges.txt',\n",
" 'dna_bpe_dict/added_tokens.json',\n",
" 'dna_bpe_dict/tokenizer.json')"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#要在 🤗 Transformers 中使用这个标记器，我们必须将它包裹在一个 PreTrainedTokenizerFast 类中\n",
"from transformers import GPT2TokenizerFast\n",
"dna_tokenizer = GPT2TokenizerFast(tokenizer_object=new_tokenizer)\n",
"dna_tokenizer.save_pretrained(\"dna_bpe_dict\")\n",
"#dna_tokenizer.push_to_hub(\"dna_bpe_dict_1g\", organization=\"dnagpt\", use_auth_token=\"hf_*****\") # push to huggingface"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "f84506d8-6208-4027-aad7-2b68a1bc16d6",
"metadata": {},
"outputs": [],
"source": [
"tokenizer_new = AutoTokenizer.from_pretrained('dna_bpe_dict')"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "d40d4d53-6fed-445c-afb5-c0346ab854c8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['TG', 'GCGTGAA', 'CCCGG', 'GATCGG', 'G']"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer_new.tokenize(\"TGGCGTGAACCCGGGATCGGG\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "640302f6-f740-41a4-ae92-ca4c43d97493",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02-gpt2_bert/2-dna-gpt.ipynb
+5167
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/3-dna-bert.ipynb
+754
View File
File diff suppressed because one or more lines are too long
02-gpt2_bert/4-gene-feature.ipynb
+453
View File
@@ -0,0 +1,453 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b1b37ca8-25a3-440c-9b68-7f72ce670ade",
"metadata": {},
"source": [
"# 2.4 基因大模型的生物序列特征提取"
]
},
{
"cell_type": "markdown",
"id": "d3d04215-2b6c-41fb-92a4-90c82d322ba4",
"metadata": {},
"source": [
"使用 GPT-2 模型获取文本的特征向量是一个常见的需求，尤其是在进行文本分类、相似度计算或其他下游任务时。Hugging Face 的 transformers 库提供了简单易用的接口来实现这一点。以下是详细的步骤和代码示例，帮助你从 GPT-2 模型中提取文本的特征向量。"
]
},
{
"cell_type": "markdown",
"id": "3ff5b7c6-e57c-4839-8510-f764154faa65",
"metadata": {},
"source": [
"使用 GPT-2 模型获取文本的特征向量是一个常见的需求，尤其是在进行文本分类、相似度计算或其他下游任务时。Hugging Face 的 `transformers` 库提供了简单易用的接口来实现这一点。以下是详细的步骤和代码示例，帮助你从 GPT-2 模型中提取文本的特征向量。\n",
"\n",
"### 方法 1: 使用隐藏状态（Hidden States）\n",
"\n",
"GPT-2 是一个基于 Transformer 的语言模型，它在每一层都有隐藏状态（hidden states），这些隐藏状态可以作为文本的特征表示。你可以选择最后一层的隐藏状态作为最终的特征向量，或者对多层的隐藏状态进行平均或拼接。\n",
"\n",
"\n",
"### 方法 2: 使用池化策略\n",
"\n",
"另一种方法是通过对所有 token 的隐藏状态进行池化操作来获得句子级别的特征向量。常见的池化方法包括：\n",
"\n",
"- **均值池化**（Mean Pooling）：对所有 token 的隐藏状态求平均。\n",
"- **最大池化**（Max Pooling）：对每个维度取最大值。"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "e7fe053b-d6da-488a-9c62-24e4b40a992d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'input_ids': tensor([[ 1, 191, 29, 753, 1241, 2104, 12297, 357, 85, 4395,\n",
" 26392, 16]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}\n",
"torch.Size([768])\n",
"torch.Size([768])\n",
"torch.Size([768])\n"
]
}
],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"tokenizer = AutoTokenizer.from_pretrained('dna_bpe_dict')\n",
"tokenizer.tokenize(\"GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG\")\n",
"#result: [G','AGCAC','ATTCGCC',....]\n",
"\n",
"model = AutoModel.from_pretrained('dna_gpt2_v0')\n",
"import torch\n",
"dna = \"ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC\"\n",
"inputs = tokenizer(dna, return_tensors = 'pt')\n",
"print(inputs)\n",
"\n",
"outputs = model(inputs[\"input_ids\"])\n",
"#outputs = model(**inputs)\n",
"\n",
"hidden_states = outputs.last_hidden_state # [1, sequence_length, 768] outputs.last_hidden_state or outputs[0]\n",
"\n",
"# embedding with mean pooling\n",
"embedding_mean = torch.mean(hidden_states[0], dim=0)\n",
"print(embedding_mean.shape) # expect to be 768\n",
"\n",
"# embedding with max pooling\n",
"embedding_max = torch.max(hidden_states[0], dim=0)[0]\n",
"print(embedding_max.shape) # expect to be 768\n",
"\n",
"# embedding with first token\n",
"embedding_first_token = hidden_states[0][0]\n",
"print(embedding_first_token.shape) # expect to be 768"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "a1f2b545-283a-4613-a953-beb82f427826",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of BertModel were not initialized from the model checkpoint at dna_bert_v0 and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'input_ids': tensor([[ 6, 200, 16057, 10, 1256, 2123, 12294, 366, 13138, 7826,\n",
" 82, 25]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}\n",
"torch.Size([768])\n",
"torch.Size([768])\n",
"torch.Size([768])\n"
]
}
],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"import torch\n",
"\n",
"tokenizer = AutoTokenizer.from_pretrained('dna_wordpiece_dict')\n",
"tokenizer.tokenize(\"GAGCACATTCGCCTGCGTGCGCACTCACACACACGTTCAAAAAGAGTCCATTCGATTCTGGCAGTAG\")\n",
"#result: [G','AGCAC','ATTCGCC',....]\n",
"\n",
"model = AutoModel.from_pretrained('dna_bert_v0')\n",
"dna = \"ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC\"\n",
"inputs = tokenizer(dna, return_tensors = 'pt')\n",
"print(inputs)\n",
"\n",
"outputs = model(inputs[\"input_ids\"])\n",
"#outputs = model(**inputs)\n",
"\n",
"hidden_states = outputs.last_hidden_state # [1, sequence_length, 768] outputs.last_hidden_state or outputs[0]\n",
"\n",
"# embedding with mean pooling\n",
"embedding_mean = torch.mean(hidden_states[0], dim=0)\n",
"print(embedding_mean.shape) # expect to be 768\n",
"\n",
"# embedding with max pooling\n",
"embedding_max = torch.max(hidden_states[0], dim=0)[0]\n",
"print(embedding_max.shape) # expect to be 768\n",
"\n",
"# embedding with first token\n",
"embedding_first_token = hidden_states[0][0]\n",
"print(embedding_first_token.shape) # expect to be 768"
]
},
{
"cell_type": "markdown",
"id": "56761874-9af7-4b90-aa8b-131e5b8c69b6",
"metadata": {},
"source": [
"## 特征提取并分类\n",
"\n",
"我们使用第一章中的\"dnagpt/dna_core_promoter\"数据集，演示下使用我们训练的DNA GPT2或者DNA bert模型，提取序列特征，然使用最基础的逻辑回归分类方法，对序列进行分类。"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f1ca177c-a80f-48a1-b2f9-16c13b3350db",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"#或者\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2295739c-e80a-47be-9400-88bfab4b0bb6",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using the latest cached version of the dataset since dnagpt/dna_core_promoter couldn't be found on the Hugging Face Hub\n",
"Found the latest cached dataset configuration 'default' at /root/.cache/huggingface/datasets/dnagpt___dna_core_promoter/default/0.0.0/809065798bf4928f67397ddba23e4aa9cc5ac3ed (last modified on Fri Dec 27 16:05:19 2024).\n"
]
},
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 59196\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset\n",
"dna_data = load_dataset(\"dnagpt/dna_core_promoter\")\n",
"dna_data"
]
},
{
"cell_type": "markdown",
"id": "c804bced-f151-43a7-8a95-156db358da3e",
"metadata": {},
"source": [
"这里，我们不需要关注这个数据的具体生物学含义，只需知道sequence是具体的DNA序列，label是分类标签，有两个类别0和1即可"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9a47a1b1-21f2-4d71-801c-50f88e326ed3",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'sequence': 'CATGCGGGTCGATATCCTATCTGAATCTCTCAGCCCAAGAGGGAGTCCGCTCATCTATTCGGCAGTACTG',\n",
" 'label': 0}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dna_data[\"train\"][0]"
]
},
{
"cell_type": "markdown",
"id": "cde7986d-a225-41ca-8f11-614d079fd2bf",
"metadata": {},
"source": [
"这里使用scikit-learn库来构建逻辑回归分类器。首先是特征提取："
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4010d991-056a-43ce-8cca-30eeec8678f5",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.metrics import accuracy_score\n",
"from transformers import GPT2Tokenizer, GPT2Model\n",
"import torch\n",
"\n",
"# 初始化 GPT-2 模型和分词器\n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
"tokenizer.pad_token = tokenizer.eos_token # 将填充符号设置为 eos_token\n",
"model = GPT2Model.from_pretrained(\"gpt2\")\n",
"\n",
"def get_gpt2_feature(sequence):\n",
" \"\"\"\n",
" 使用 GPT-2 模型提取特征向量。\n",
" :param sequence: DNA 序列 (字符串格式)\n",
" :return: 平均特征向量 (numpy 数组)\n",
" \"\"\"\n",
" # 将 DNA 序列分词并转换为 GPT-2 输入\n",
" inputs = tokenizer(sequence, return_tensors=\"pt\", padding=True, truncation=True)\n",
" with torch.no_grad():\n",
" outputs = model(**inputs)\n",
" # 提取最后一层的隐藏状态作为特征向量并平均，会对每个序列的所有 token 的特征进行平均，最终得到一个形状为 (1, 768) 的向量（对于 batch_size=1）\n",
" feature_vector = outputs.last_hidden_state.mean(dim=1).detach().numpy()\n",
" return feature_vector\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "057eee1e-9f9a-47a2-b577-588caec58d31",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Processing DNA data: 100%|██████████| 59196/59196 [25:16<00:00, 39.04it/s]\n"
]
}
],
"source": [
"from tqdm import tqdm\n",
"# 提取特征和标签\n",
"X = []\n",
"Y = []\n",
"\n",
"# 存储特征向量和标签\n",
"for item in tqdm(dna_data[\"train\"], desc=\"Processing DNA data\"):\n",
" sequence = item[\"sequence\"]\n",
" label = item[\"label\"]\n",
" x_v = get_gpt2_feature(sequence)\n",
" y_v = label\n",
" X.append(x_v)\n",
" Y.append(y_v)\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "51133c2a-42e7-4e11-a6f9-6812a4e54182",
"metadata": {},
"outputs": [],
"source": [
"X = np.array(X).squeeze(1) # 去掉维度为1的那一维"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "5ab0c188-6476-43c4-b361-a2bfe0ec7a8a",
"metadata": {},
"outputs": [],
"source": [
"# 将数据分为训练集和测试集\n",
"X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42)\n",
"\n",
"# 创建逻辑回归模型\n",
"model = LogisticRegression(max_iter=200, solver='newton-cg')\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "fd9be2bf-331e-4905-99e6-832e58a0463a",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Training Logistic Regression: 100%|██████████| 200/200 [27:45<00:00, 8.33s/it]\n"
]
}
],
"source": [
"# 训练模型\n",
"for i in tqdm(range(200), desc=\"Training Logistic Regression\"):\n",
" model.fit(X_train, y_train)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "5417e4e2-3bca-4718-83a1-f418ad8a65b6",
"metadata": {},
"outputs": [],
"source": [
"# 在测试集上进行预测\n",
"y_pred = model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "34371f07-0cbe-43cf-99a1-2ccd55e43e14",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 77.48%\n"
]
}
],
"source": [
"# 计算准确率\n",
"accuracy = accuracy_score(y_test, y_pred)\n",
"print(f\"Accuracy: {accuracy * 100:.2f}%\")"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "15472a1e-7813-4ccd-878b-e0cf5d7ce095",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"True: 0, Predicted: 0\n",
"True: 0, Predicted: 1\n",
"True: 1, Predicted: 1\n",
"True: 0, Predicted: 0\n",
"True: 0, Predicted: 0\n"
]
}
],
"source": [
"# 输出部分预测结果与真实标签对比\n",
"for i in range(5):\n",
" print(f\"True: {y_test[i]}, Predicted: {y_pred[i]}\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02-gpt2_bert/5-multi-seq-gpt.ipynb
+13106
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_bpe_dict.json
+150015
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/merges-checkpoint.txt
+29995
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/special_tokens_map-checkpoint.json
+5
View File
@@ -0,0 +1,5 @@
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"unk_token": "<|endoftext|>"
}
02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/tokenizer-checkpoint.json
+150015
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_bpe_dict/.ipynb_checkpoints/tokenizer_config-checkpoint.json
+20
View File
@@ -0,0 +1,20 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"0": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<|endoftext|>",
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"extra_special_tokens": {},
"model_max_length": 1000000000000000019884624838656,
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}
02-gpt2_bert/dna_bpe_dict/merges.txt
+29995
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_bpe_dict/special_tokens_map.json
+5
View File
@@ -0,0 +1,5 @@
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"unk_token": "<|endoftext|>"
}
02-gpt2_bert/dna_bpe_dict/tokenizer.json
+150015
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_bpe_dict/tokenizer_config.json
+20
View File
@@ -0,0 +1,20 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"0": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<|endoftext|>",
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"extra_special_tokens": {},
"model_max_length": 1000000000000000019884624838656,
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}
02-gpt2_bert/dna_bpe_dict/vocab.json
+1
View File
File diff suppressed because one or more lines are too long
02-gpt2_bert/dna_wordpiece_dict.json
+30064
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_wordpiece_dict/.ipynb_checkpoints/special_tokens_map-checkpoint.json
+7
View File
@@ -0,0 +1,7 @@
{
"cls_token": "[CLS]",
"mask_token": "[MASK]",
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"unk_token": "[UNK]"
}
02-gpt2_bert/dna_wordpiece_dict/.ipynb_checkpoints/tokenizer-checkpoint.json
+30064
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_wordpiece_dict/.ipynb_checkpoints/tokenizer_config-checkpoint.json
+53
View File
@@ -0,0 +1,53 @@
{
"added_tokens_decoder": {
"0": {
"content": "[PAD]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "[UNK]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "[CLS]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "[SEP]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"4": {
"content": "[MASK]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"clean_up_tokenization_spaces": false,
"cls_token": "[CLS]",
"extra_special_tokens": {},
"mask_token": "[MASK]",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"tokenizer_class": "PreTrainedTokenizerFast",
"unk_token": "[UNK]"
}
02-gpt2_bert/dna_wordpiece_dict/special_tokens_map.json
+7
View File
@@ -0,0 +1,7 @@
{
"cls_token": "[CLS]",
"mask_token": "[MASK]",
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"unk_token": "[UNK]"
}
02-gpt2_bert/dna_wordpiece_dict/tokenizer.json
+30064
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/dna_wordpiece_dict/tokenizer_config.json
+53
View File
@@ -0,0 +1,53 @@
{
"added_tokens_decoder": {
"0": {
"content": "[PAD]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "[UNK]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "[CLS]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "[SEP]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"4": {
"content": "[MASK]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"clean_up_tokenization_spaces": false,
"cls_token": "[CLS]",
"extra_special_tokens": {},
"mask_token": "[MASK]",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"tokenizer_class": "PreTrainedTokenizerFast",
"unk_token": "[UNK]"
}
02-gpt2_bert/env_ini.ipynb
+267
View File
@@ -0,0 +1,267 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 5,
"id": "e3fbdac5-cd38-4e41-b5d2-d9d112b4ac1b",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
"Requirement already satisfied: transformers in /root/miniconda3/lib/python3.12/site-packages (4.47.1)\n",
"Requirement already satisfied: sentencepiece in /root/miniconda3/lib/python3.12/site-packages (0.2.0)\n",
"Requirement already satisfied: google in /root/miniconda3/lib/python3.12/site-packages (3.0.0)\n",
"Requirement already satisfied: protobuf in /root/miniconda3/lib/python3.12/site-packages (5.27.0)\n",
"Requirement already satisfied: deepspeed in /root/miniconda3/lib/python3.12/site-packages (0.16.2)\n",
"Requirement already satisfied: peft in /root/miniconda3/lib/python3.12/site-packages (0.14.0)\n",
"Collecting datasets\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d7/84/0df6c5981f5fc722381662ff8cfbdf8aad64bec875f75d80b55bfef394ce/datasets-3.2.0-py3-none-any.whl (480 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m480.6/480.6 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: filelock in /root/miniconda3/lib/python3.12/site-packages (from transformers) (3.14.0)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.27.0)\n",
"Requirement already satisfied: numpy>=1.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (1.26.4)\n",
"Requirement already satisfied: packaging>=20.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (23.2)\n",
"Requirement already satisfied: pyyaml>=5.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (6.0.1)\n",
"Requirement already satisfied: regex!=2019.12.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2024.11.6)\n",
"Requirement already satisfied: requests in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2.31.0)\n",
"Requirement already satisfied: tokenizers<0.22,>=0.21 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.21.0)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.4.5)\n",
"Requirement already satisfied: tqdm>=4.27 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (4.66.2)\n",
"Requirement already satisfied: beautifulsoup4 in /root/miniconda3/lib/python3.12/site-packages (from google) (4.12.3)\n",
"Requirement already satisfied: einops in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (0.8.0)\n",
"Requirement already satisfied: hjson in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (3.1.0)\n",
"Requirement already satisfied: msgpack in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.1.0)\n",
"Requirement already satisfied: ninja in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.11.1.3)\n",
"Requirement already satisfied: psutil in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (5.9.8)\n",
"Requirement already satisfied: py-cpuinfo in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (9.0.0)\n",
"Requirement already satisfied: pydantic>=2.0.0 in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.10.4)\n",
"Requirement already satisfied: torch in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.3.0+cu121)\n",
"Requirement already satisfied: nvidia-ml-py in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (12.560.30)\n",
"Requirement already satisfied: accelerate>=0.21.0 in /root/miniconda3/lib/python3.12/site-packages (from peft) (1.2.1)\n",
"Collecting pyarrow>=15.0.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/3a/2e/3b99f8a3d9e0ccae0e961978a0d0089b25fb46ebbcfb5ebae3cca179a5b3/pyarrow-18.1.0-cp312-cp312-manylinux_2_28_x86_64.whl (40.1 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.1/40.1 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting dill<0.3.9,>=0.3.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/c9/7a/cef76fd8438a42f96db64ddaa85280485a9c395e7df3db8158cfec1eee34/dill-0.3.8-py3-none-any.whl (116 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m53.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting pandas (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/38/f8/d8fddee9ed0d0c0f4a2132c1dfcf0e3e53265055da8df952a53e7eaf178c/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.7/12.7 MB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting requests (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tqdm>=4.27 (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting xxhash (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/a7/81dba5010f7e733de88af9555725146fc133be97ce36533867f4c7e75066/xxhash-3.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.4/194.4 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multiprocess<0.70.17 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/0a/7d/a988f258104dcd2ccf1ed40fdc97e26c4ac351eeaf81d76e266c52d84e2f/multiprocess-0.70.16-py312-none-any.whl (146 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.7/146.7 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /root/miniconda3/lib/python3.12/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.5.0)\n",
"Collecting aiohttp (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/40/7f/6de218084f9b653026bd7063cd8045123a7ba90c25176465f266976d8c82/aiohttp-3.11.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting aiohappyeyeballs>=2.3.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/b9/74/fbb6559de3607b3300b9be3cc64e97548d55678e44623db17820dbd20002/aiohappyeyeballs-2.4.4-py3-none-any.whl (14 kB)\n",
"Collecting aiosignal>=1.1.2 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/ec/6a/bc7e17a3e87a2985d3e8f4da4cd0f481060eb78fb08596c42be62c90a4d9/aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)\n",
"Requirement already satisfied: attrs>=17.3.0 in /root/miniconda3/lib/python3.12/site-packages (from aiohttp->datasets) (23.2.0)\n",
"Collecting frozenlist>=1.1.1 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/af/f2/64b73a9bb86f5a89fb55450e97cd5c1f84a862d4ff90d9fd1a73ab0f64a5/frozenlist-1.5.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m283.6/283.6 kB\u001b[0m \u001b[31m41.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multidict<7.0,>=4.5 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d3/c8/529101d7176fe7dfe1d99604e48d69c5dfdcadb4f06561f465c8ef12b4df/multidict-6.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (131 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m131.0/131.0 kB\u001b[0m \u001b[31m56.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting propcache>=0.2.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1c/07/ebe102777a830bca91bbb93e3479cd34c2ca5d0361b83be9dbd93104865e/propcache-0.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (243 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.6/243.6 kB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting yarl<2.0,>=1.17.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1a/e1/a097d5755d3ea8479a42856f51d97eeff7a3a7160593332d98f2709b3580/yarl-1.18.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (336 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m336.9/336.9 kB\u001b[0m \u001b[31m41.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /root/miniconda3/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.24.0->transformers) (4.12.2)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.2 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (2.27.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.0.4)\n",
"Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (3.7)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.1.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2024.2.2)\n",
"Requirement already satisfied: sympy in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (1.12.1)\n",
"Requirement already satisfied: networkx in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.3)\n",
"Requirement already satisfied: jinja2 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.1.4)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (8.9.2.26)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.3.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.0.2.54)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (10.3.2.106)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.4.5.107)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.0.106)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (2.20.5)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/miniconda3/lib/python3.12/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->deepspeed) (12.5.40)\n",
"Requirement already satisfied: soupsieve>1.2 in /root/miniconda3/lib/python3.12/site-packages (from beautifulsoup4->google) (2.5)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /root/miniconda3/lib/python3.12/site-packages (from pandas->datasets) (2.9.0.post0)\n",
"Collecting pytz>=2020.1 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/c3/005fcca25ce078d2cc29fd559379817424e94885510568bc1bc53d7d5846/pytz-2024.2-py2.py3-none-any.whl (508 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m508.0/508.0 kB\u001b[0m \u001b[31m38.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tzdata>=2022.7 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/a6/ab/7e5f53c3b9d14972843a647d8d7a853969a58aecc7559cb3267302c94774/tzdata-2024.2-py2.py3-none-any.whl (346 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m346.6/346.6 kB\u001b[0m \u001b[31m36.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: six>=1.5 in /root/miniconda3/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/lib/python3.12/site-packages (from jinja2->torch->deepspeed) (2.1.5)\n",
"Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /root/miniconda3/lib/python3.12/site-packages (from sympy->torch->deepspeed) (1.3.0)\n",
"Installing collected packages: pytz, xxhash, tzdata, tqdm, requests, pyarrow, propcache, multidict, frozenlist, dill, aiohappyeyeballs, yarl, pandas, multiprocess, aiosignal, aiohttp, datasets\n",
" Attempting uninstall: tqdm\n",
" Found existing installation: tqdm 4.66.2\n",
" Uninstalling tqdm-4.66.2:\n",
" Successfully uninstalled tqdm-4.66.2\n",
" Attempting uninstall: requests\n",
" Found existing installation: requests 2.31.0\n",
" Uninstalling requests-2.31.0:\n",
" Successfully uninstalled requests-2.31.0\n",
"Successfully installed aiohappyeyeballs-2.4.4 aiohttp-3.11.11 aiosignal-1.3.2 datasets-3.2.0 dill-0.3.8 frozenlist-1.5.0 multidict-6.1.0 multiprocess-0.70.16 pandas-2.2.3 propcache-0.2.1 pyarrow-18.1.0 pytz-2024.2 requests-2.32.3 tqdm-4.67.1 tzdata-2024.2 xxhash-3.5.0 yarl-1.18.3\n",
"\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"!pip install transformers sentencepiece google protobuf deepspeed peft datasets "
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "4e906370-40c7-4f6b-a700-f183a9308c78",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"https://hf-mirror.com\n"
]
}
],
"source": [
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ecc98529-6581-41d2-a876-23ce5187cae7",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b01fc372-33af-46e5-8c0e-8bccba7237ee",
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"# load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"dnagpt/dna_core_promoter\")['train'].train_test_split(test_size=0.1)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "136c38d4-bd0f-4ecd-9165-2fd5b5207c1d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 53276\n",
" })\n",
" test: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 5920\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "markdown",
"id": "28acb64e-8d1e-4482-a515-344a2e0344c4",
"metadata": {},
"source": [
"## lfs 支持\n",
"apt-get update\n",
"\n",
"apt-get install git-lfs\n",
"\n",
"git lfs install"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d3cefb0-1eed-4f23-8591-1990f7113820",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
02-gpt2_bert/gene_en_bpe.py
+19
View File
@@ -0,0 +1,19 @@
from tokenizers import (
decoders,
models,
normalizers,
pre_tokenizers,
processors,
trainers,
Tokenizer,
)
tokenizer = Tokenizer(models.BPE())
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False, use_regex=False) #use_regex=False,空格当成一般字符串
trainer = trainers.BpeTrainer(vocab_size=90000, special_tokens=["<|endoftext|>"]) #9w words
tokenizer.train(["dna_1g.txt","protein_1g.txt","english_500m.txt"]
, trainer=trainer) #all file list, take 10-20 min
tokenizer.save("gene_eng_dict.json")
02-gpt2_bert/gene_eng_dict.json
+449219
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/gene_eng_dict/merges.txt
+89796
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/gene_eng_dict/special_tokens_map.json
+5
View File
@@ -0,0 +1,5 @@
{
"bos_token": "<|endoftext|>",
"eos_token": "<|endoftext|>",
"unk_token": "<|endoftext|>"
}
02-gpt2_bert/gene_eng_dict/tokenizer.json
+449219
View File
File diff suppressed because it is too large Load Diff
02-gpt2_bert/gene_eng_dict/tokenizer_config.json
+20
View File
@@ -0,0 +1,20 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"0": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<|endoftext|>",
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"extra_special_tokens": {},
"model_max_length": 1000000000000000019884624838656,
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}
02-gpt2_bert/gene_eng_dict/vocab.json
+1
View File
File diff suppressed because one or more lines are too long
02-gpt2_bert/img/.ipynb_checkpoints/gpt2-stru-checkpoint.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 342 KiB

02-gpt2_bert/img/gpt2-netron.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

02-gpt2_bert/img/gpt2-stru.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 342 KiB

02-gpt2_bert/img/llm-visual.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 574 KiB

03-gene-task/.ipynb_checkpoints/1-category-task-checkpoint.ipynb
+812
View File
File diff suppressed because one or more lines are too long
03-gene-task/.ipynb_checkpoints/3-multi-seq-task-checkpoint.ipynb
+863
View File
File diff suppressed because one or more lines are too long
03-gene-task/.ipynb_checkpoints/5-regression-task-checkpoint.ipynb
+587
View File
@@ -0,0 +1,587 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c499a5c3-0244-41c4-9947-e166206204e2",
"metadata": {},
"source": [
"# 3.5 回归类任务"
]
},
{
"cell_type": "markdown",
"id": "4678171b-bbc8-49dd-ad04-48f5ef89b45e",
"metadata": {},
"source": [
"GPT-2 原本是设计用于生成自然语言的模型，但通过适当的调整和微调，它也可以用于回归任务，例如预测连续值。\n",
"\n",
"使用 GPT-2 进行回归问题的解决，可以将回归问题转化为自回归语言模型任务。GPT-2 原本是设计用于生成自然语言的模型，但通过适当的调整和微调，它也可以用于回归任务，例如预测连续值（如情感评分、价格预测等）。\n",
"\n",
"---\n",
"\n",
"### **1. 使用 GPT-2 做回归的核心思路**\n",
"\n",
"1. **调整输出层**：\n",
" - 默认情况下，GPT-2 的输出是一个词汇表大小的概率分布，用于预测下一个 token。\n",
" - 对于回归问题，可以将模型的最后一层替换为一个线性层，使得输出变为一个标量或多个连续值。\n",
" - gpt2的huggingface实现中，可以简单设置1个分类的分类header，实现回归预测。\n",
"\n",
"2. **损失函数**：\n",
" - 对于回归问题，使用均方误差（MSE）或均绝对误差（MAE）作为损失函数，而不是分类任务中常用的交叉熵。\n",
"\n",
"3. **输入格式**：\n",
" - 输入数据仍然是文本，可以通过特定的模板形式加入上下文信息。\n",
"\n",
"---\n",
"\n",
"### **2. GPT-2 回归任务的实现步骤**\n",
"\n",
"#### **（1）加载基础模型**\n",
"\n",
"从 Hugging Face Transformers 库加载 GPT-2 模型和分词器，并调整其配置以适应回归任务。\n",
"\n",
"```python\n",
"from transformers import GPT2Tokenizer, GPT2Model, GPT2Config, AutoModelForSequenceClassification\n",
"\n",
"# 加载分词器\n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
"\n",
"# 调整模型配置，num_labels=1 表示回归任务\n",
"config = GPT2Config.from_pretrained(\"gpt2\", num_labels=1)\n",
"\n",
"# 加载模型，增加回归输出\n",
"model = AutoModelForSequenceClassification.from_pretrained(\"gpt2\", config=config)\n",
"```\n",
"\n",
"---\n",
"\n",
"### **3. 课程数据集**\n",
"\n",
"本例程使用了蛋白质稳定性分析的数据集，也就是一个蛋白质序列，对应一个float的数值，做回归预测分析。\n",
"\n",
"**蛋白质稳定性分析**是研究蛋白质在不同条件下保持其结构和功能的能力的过程。蛋白质稳定性是生物化学和生物技术领域的重要课题，影响着蛋白质的折叠、功能执行、以及在应用中的可用性（如工业酶、药物开发等）。\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "1e8c0f86-af78-43e1-8db4-e2a2ea22f815",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c51a8d69-9a36-47e7-8084-f64e6a72e4f7",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"from tokenizers import Tokenizer\n",
"from transformers import GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
"from transformers import AutoModelForSequenceClassification\n",
"from transformers import DataCollatorWithPadding"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a5aeb7c1-2d2a-4f57-ad8c-659613870e59",
"metadata": {},
"outputs": [],
"source": [
"#set tokenizer\n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"dnagpt/gene_eng_gpt2_v0\")\n",
"tokenizer.pad_token = tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ad0c19cd-96a5-463e-8b7d-439646fef429",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at dnagpt/gene_eng_gpt2_v0 and are newly initialized: ['score.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"#set model\n",
"model = AutoModelForSequenceClassification.from_pretrained('dnagpt/gene_eng_gpt2_v0', num_labels=1)\n",
"model.config.pad_token_id = model.config.eos_token_id"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "8c48cb0a-6142-4afc-823e-08fb33f74222",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['seq_id', 'seq_type', 'seq', 'label'],\n",
" num_rows: 62079\n",
" })\n",
" test: Dataset({\n",
" features: ['seq_id', 'seq_type', 'seq', 'label'],\n",
" num_rows: 6898\n",
" })\n",
"})"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset\n",
"# 1. load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"csv\", data_files=\"data/protein_stab.csv\")['train'].train_test_split(test_size=0.1)\n",
"dataset"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "685dd025-f00a-4869-bc30-9843c77b6d8a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'seq_id': 'train_prot_32672',\n",
" 'seq_type': 'prot',\n",
" 'seq': 'FYRLIIFKYPDYIDTYLRLAAIAKEKNNLQLSIEGNGSGGNGSGGNGSGN',\n",
" 'label': 0.7599999904632561}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset[\"train\"][0]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6e10dbbb-73ef-4b67-8290-77f8896298f5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"datasets mean token lenght 17.24006958538707 min token length 12 max token length 35\n"
]
}
],
"source": [
"token_len_list = []\n",
"for item in dataset[\"test\"]:\n",
" inputs = tokenizer.tokenize(item[\"seq\"])\n",
" token_len_list.append( len(inputs) )\n",
"\n",
"mean_len = sum(token_len_list)/len(token_len_list)\n",
"min_len = min(token_len_list)\n",
"max_len = max(token_len_list)\n",
"\n",
"print(\"datasets \", \"mean token lenght\", mean_len, \"min token length\", min_len, \"max token length\", max_len)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "ac58b5b4-bff0-404d-bcf5-2b93db2b37c0",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "419cce8c5ba249ac8c8773dd2d69992d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/62079 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Asking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.\n",
"Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0b9ea09fe3ea49b19f7d52aca7949acf",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/6898 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# 2. tokenize\n",
"def tokenize_function(examples):\n",
" return tokenizer(examples['seq'], truncation=True, padding='max_length')\n",
"\n",
"# 3. 对数据集应用分词函数\n",
"tokenized_datasets = dataset.map(tokenize_function, batched=True)\n",
"\n",
"# 4. 创建一个数据收集器，用于动态填充和遮蔽\n",
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "94f6d643-2cf7-4651-9a8d-1884b2bddd1c",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/transformers/training_args.py:1575: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
" warnings.warn(\n",
"/tmp/ipykernel_1347/4285456223.py:23: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n",
" trainer = Trainer(\n"
]
}
],
"source": [
"from transformers import TrainingArguments, Trainer\n",
"import numpy as np\n",
"from sklearn.metrics import mean_squared_error\n",
"\n",
"\n",
"def compute_metrics(eval_pred):\n",
" predictions, labels = eval_pred\n",
" rmse = mean_squared_error(labels, predictions)\n",
" return {\"rmse\": rmse}\n",
"\n",
"# 设置训练参数\n",
"training_args = TrainingArguments(\n",
" output_dir='./results',\n",
" evaluation_strategy=\"epoch\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=20,\n",
" per_device_eval_batch_size=20,\n",
" num_train_epochs=10,\n",
" weight_decay=0.01,\n",
")\n",
"\n",
"# 使用Trainer API进行训练（假设已有train_dataset和eval_dataset）\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"test\"],\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dfe12979-d977-4404-bf9e-18c1f91a3e39",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='30987' max='31040' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [30987/31040 1:00:56 < 00:06, 8.47 it/s, Epoch 9.98/10]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" <th>Rmse</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.044600</td>\n",
" <td>0.163462</td>\n",
" <td>0.163462</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.041900</td>\n",
" <td>0.157900</td>\n",
" <td>0.157900</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.037700</td>\n",
" <td>0.159724</td>\n",
" <td>0.159724</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>0.031700</td>\n",
" <td>0.157686</td>\n",
" <td>0.157686</td>\n",
" </tr>\n",
" <tr>\n",
" <td>5</td>\n",
" <td>0.028800</td>\n",
" <td>0.157124</td>\n",
" <td>0.157124</td>\n",
" </tr>\n",
" <tr>\n",
" <td>6</td>\n",
" <td>0.025400</td>\n",
" <td>0.150852</td>\n",
" <td>0.150852</td>\n",
" </tr>\n",
" <tr>\n",
" <td>7</td>\n",
" <td>0.022300</td>\n",
" <td>0.159293</td>\n",
" <td>0.159293</td>\n",
" </tr>\n",
" <tr>\n",
" <td>8</td>\n",
" <td>0.019600</td>\n",
" <td>0.154608</td>\n",
" <td>0.154608</td>\n",
" </tr>\n",
" <tr>\n",
" <td>9</td>\n",
" <td>0.017300</td>\n",
" <td>0.156104</td>\n",
" <td>0.156104</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"IOPub message rate exceeded.\n",
"The Jupyter server will temporarily stop sending output\n",
"to the client in order to avoid crashing it.\n",
"To change this limit, set the config variable\n",
"`--ServerApp.iopub_msg_rate_limit`.\n",
"\n",
"Current values:\n",
"ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
"ServerApp.rate_limit_window=3.0 (secs)\n",
"\n"
]
}
],
"source": [
"# 开始训练\n",
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "060c4618-40d0-4934-bab8-36aab3a46de5",
"metadata": {},
"outputs": [],
"source": [
"#模型测试\n",
"predictions = trainer.predict(tokenized_datasets[\"test\"])\n",
"predictions"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "1f8ef885-5bc9-4668-905b-6b2235209654",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='345' max='345' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [345/345 00:09]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.15949687361717224,\n",
" 'eval_rmse': 0.15949687361717224,\n",
" 'eval_runtime': 9.1483,\n",
" 'eval_samples_per_second': 754.017,\n",
" 'eval_steps_per_second': 37.712,\n",
" 'epoch': 10.0}"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.evaluate()"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "afabdbe9-9b96-4f9e-bef2-1d819431f8d1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[ 1.7208484 ]\n",
" [ 0.00225139]\n",
" [ 0.3325616 ]\n",
" [-0.34372616]\n",
" [-0.45505935]\n",
" [-0.06892765]\n",
" [ 0.15099108]\n",
" [ 0.12211376]\n",
" [ 0.3947332 ]\n",
" [ 0.23186803]]\n"
]
}
],
"source": [
"predictions.predictions[0:10].squeeze()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "fa9d17fd-eece-4c1e-99e0-3d19d36f7584",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 1.69, 0.84, 0.58, -0.15, 0.23, 0.03, 0.15, 0.2 , 0.51,\n",
" 1.1 ], dtype=float32)"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predictions.label_ids[0:10]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "52252015-e068-414b-bd8a-79a5d1a2beec",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03-gene-task/.ipynb_checkpoints/env_ini-checkpoint.ipynb
+267
View File
@@ -0,0 +1,267 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 5,
"id": "e3fbdac5-cd38-4e41-b5d2-d9d112b4ac1b",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
"Requirement already satisfied: transformers in /root/miniconda3/lib/python3.12/site-packages (4.47.1)\n",
"Requirement already satisfied: sentencepiece in /root/miniconda3/lib/python3.12/site-packages (0.2.0)\n",
"Requirement already satisfied: google in /root/miniconda3/lib/python3.12/site-packages (3.0.0)\n",
"Requirement already satisfied: protobuf in /root/miniconda3/lib/python3.12/site-packages (5.27.0)\n",
"Requirement already satisfied: deepspeed in /root/miniconda3/lib/python3.12/site-packages (0.16.2)\n",
"Requirement already satisfied: peft in /root/miniconda3/lib/python3.12/site-packages (0.14.0)\n",
"Collecting datasets\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d7/84/0df6c5981f5fc722381662ff8cfbdf8aad64bec875f75d80b55bfef394ce/datasets-3.2.0-py3-none-any.whl (480 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m480.6/480.6 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: filelock in /root/miniconda3/lib/python3.12/site-packages (from transformers) (3.14.0)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.27.0)\n",
"Requirement already satisfied: numpy>=1.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (1.26.4)\n",
"Requirement already satisfied: packaging>=20.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (23.2)\n",
"Requirement already satisfied: pyyaml>=5.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (6.0.1)\n",
"Requirement already satisfied: regex!=2019.12.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2024.11.6)\n",
"Requirement already satisfied: requests in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2.31.0)\n",
"Requirement already satisfied: tokenizers<0.22,>=0.21 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.21.0)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.4.5)\n",
"Requirement already satisfied: tqdm>=4.27 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (4.66.2)\n",
"Requirement already satisfied: beautifulsoup4 in /root/miniconda3/lib/python3.12/site-packages (from google) (4.12.3)\n",
"Requirement already satisfied: einops in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (0.8.0)\n",
"Requirement already satisfied: hjson in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (3.1.0)\n",
"Requirement already satisfied: msgpack in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.1.0)\n",
"Requirement already satisfied: ninja in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.11.1.3)\n",
"Requirement already satisfied: psutil in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (5.9.8)\n",
"Requirement already satisfied: py-cpuinfo in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (9.0.0)\n",
"Requirement already satisfied: pydantic>=2.0.0 in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.10.4)\n",
"Requirement already satisfied: torch in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.3.0+cu121)\n",
"Requirement already satisfied: nvidia-ml-py in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (12.560.30)\n",
"Requirement already satisfied: accelerate>=0.21.0 in /root/miniconda3/lib/python3.12/site-packages (from peft) (1.2.1)\n",
"Collecting pyarrow>=15.0.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/3a/2e/3b99f8a3d9e0ccae0e961978a0d0089b25fb46ebbcfb5ebae3cca179a5b3/pyarrow-18.1.0-cp312-cp312-manylinux_2_28_x86_64.whl (40.1 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.1/40.1 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting dill<0.3.9,>=0.3.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/c9/7a/cef76fd8438a42f96db64ddaa85280485a9c395e7df3db8158cfec1eee34/dill-0.3.8-py3-none-any.whl (116 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m53.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting pandas (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/38/f8/d8fddee9ed0d0c0f4a2132c1dfcf0e3e53265055da8df952a53e7eaf178c/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.7/12.7 MB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting requests (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tqdm>=4.27 (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting xxhash (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/a7/81dba5010f7e733de88af9555725146fc133be97ce36533867f4c7e75066/xxhash-3.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.4/194.4 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multiprocess<0.70.17 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/0a/7d/a988f258104dcd2ccf1ed40fdc97e26c4ac351eeaf81d76e266c52d84e2f/multiprocess-0.70.16-py312-none-any.whl (146 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.7/146.7 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /root/miniconda3/lib/python3.12/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.5.0)\n",
"Collecting aiohttp (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/40/7f/6de218084f9b653026bd7063cd8045123a7ba90c25176465f266976d8c82/aiohttp-3.11.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting aiohappyeyeballs>=2.3.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/b9/74/fbb6559de3607b3300b9be3cc64e97548d55678e44623db17820dbd20002/aiohappyeyeballs-2.4.4-py3-none-any.whl (14 kB)\n",
"Collecting aiosignal>=1.1.2 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/ec/6a/bc7e17a3e87a2985d3e8f4da4cd0f481060eb78fb08596c42be62c90a4d9/aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)\n",
"Requirement already satisfied: attrs>=17.3.0 in /root/miniconda3/lib/python3.12/site-packages (from aiohttp->datasets) (23.2.0)\n",
"Collecting frozenlist>=1.1.1 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/af/f2/64b73a9bb86f5a89fb55450e97cd5c1f84a862d4ff90d9fd1a73ab0f64a5/frozenlist-1.5.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m283.6/283.6 kB\u001b[0m \u001b[31m41.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multidict<7.0,>=4.5 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d3/c8/529101d7176fe7dfe1d99604e48d69c5dfdcadb4f06561f465c8ef12b4df/multidict-6.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (131 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m131.0/131.0 kB\u001b[0m \u001b[31m56.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting propcache>=0.2.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1c/07/ebe102777a830bca91bbb93e3479cd34c2ca5d0361b83be9dbd93104865e/propcache-0.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (243 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.6/243.6 kB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting yarl<2.0,>=1.17.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1a/e1/a097d5755d3ea8479a42856f51d97eeff7a3a7160593332d98f2709b3580/yarl-1.18.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (336 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m336.9/336.9 kB\u001b[0m \u001b[31m41.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /root/miniconda3/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.24.0->transformers) (4.12.2)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.2 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (2.27.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.0.4)\n",
"Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (3.7)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.1.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2024.2.2)\n",
"Requirement already satisfied: sympy in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (1.12.1)\n",
"Requirement already satisfied: networkx in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.3)\n",
"Requirement already satisfied: jinja2 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.1.4)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (8.9.2.26)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.3.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.0.2.54)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (10.3.2.106)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.4.5.107)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.0.106)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (2.20.5)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/miniconda3/lib/python3.12/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->deepspeed) (12.5.40)\n",
"Requirement already satisfied: soupsieve>1.2 in /root/miniconda3/lib/python3.12/site-packages (from beautifulsoup4->google) (2.5)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /root/miniconda3/lib/python3.12/site-packages (from pandas->datasets) (2.9.0.post0)\n",
"Collecting pytz>=2020.1 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/c3/005fcca25ce078d2cc29fd559379817424e94885510568bc1bc53d7d5846/pytz-2024.2-py2.py3-none-any.whl (508 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m508.0/508.0 kB\u001b[0m \u001b[31m38.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tzdata>=2022.7 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/a6/ab/7e5f53c3b9d14972843a647d8d7a853969a58aecc7559cb3267302c94774/tzdata-2024.2-py2.py3-none-any.whl (346 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m346.6/346.6 kB\u001b[0m \u001b[31m36.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: six>=1.5 in /root/miniconda3/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/lib/python3.12/site-packages (from jinja2->torch->deepspeed) (2.1.5)\n",
"Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /root/miniconda3/lib/python3.12/site-packages (from sympy->torch->deepspeed) (1.3.0)\n",
"Installing collected packages: pytz, xxhash, tzdata, tqdm, requests, pyarrow, propcache, multidict, frozenlist, dill, aiohappyeyeballs, yarl, pandas, multiprocess, aiosignal, aiohttp, datasets\n",
" Attempting uninstall: tqdm\n",
" Found existing installation: tqdm 4.66.2\n",
" Uninstalling tqdm-4.66.2:\n",
" Successfully uninstalled tqdm-4.66.2\n",
" Attempting uninstall: requests\n",
" Found existing installation: requests 2.31.0\n",
" Uninstalling requests-2.31.0:\n",
" Successfully uninstalled requests-2.31.0\n",
"Successfully installed aiohappyeyeballs-2.4.4 aiohttp-3.11.11 aiosignal-1.3.2 datasets-3.2.0 dill-0.3.8 frozenlist-1.5.0 multidict-6.1.0 multiprocess-0.70.16 pandas-2.2.3 propcache-0.2.1 pyarrow-18.1.0 pytz-2024.2 requests-2.32.3 tqdm-4.67.1 tzdata-2024.2 xxhash-3.5.0 yarl-1.18.3\n",
"\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"!pip install transformers sentencepiece google protobuf deepspeed peft datasets "
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "4e906370-40c7-4f6b-a700-f183a9308c78",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"https://hf-mirror.com\n"
]
}
],
"source": [
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ecc98529-6581-41d2-a876-23ce5187cae7",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b01fc372-33af-46e5-8c0e-8bccba7237ee",
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"# load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"dnagpt/dna_core_promoter\")['train'].train_test_split(test_size=0.1)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "136c38d4-bd0f-4ecd-9165-2fd5b5207c1d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 53276\n",
" })\n",
" test: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 5920\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "markdown",
"id": "28acb64e-8d1e-4482-a515-344a2e0344c4",
"metadata": {},
"source": [
"## lfs 支持\n",
"apt-get update\n",
"\n",
"apt-get install git-lfs\n",
"\n",
"git lfs install"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d3cefb0-1eed-4f23-8591-1990f7113820",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03-gene-task/1-category-task.ipynb
+812
View File
File diff suppressed because one or more lines are too long
03-gene-task/2-structure-predict.ipynb
+986
View File
@@ -0,0 +1,986 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "212e1052-e0d9-404f-a4ee-db199a4c6d17",
"metadata": {},
"source": [
"# 3.2 序列结构预测"
]
},
{
"cell_type": "markdown",
"id": "0eb5d83c-8dd6-498b-adc9-1f74c97c3427",
"metadata": {},
"source": [
"蛋白质的结构可分为四级：\n",
"\n",
"1. 一级结构也就是氨基酸序列；\n",
"2. 二级结构是周期性的结构构象，比如α螺旋β折叠等\n",
"3. 三级结构是整条多肽链的三维空间结构\n",
"4. 四级结构是几个蛋白质分子形成的复合体结构，比如三聚体，四聚 体等\n",
"\n",
"\n",
"二级结构（Secondary Structure）是指生物大分子如蛋白质和核酸（RNA 和 DNA）中局部的、有规则的空间构象。这些结构是由分子内的一些化学键或相互作用稳定下来的，但不涉及整个分子的整体折叠状态。以下是关于蛋白质和 RNA 二级结构的简单介绍：\n",
"\n",
"### 蛋白质的二级结构\n",
"\n",
"蛋白质的二级结构主要由主链原子间的氢键形成，具体包括以下几种常见的类型：\n",
"\n",
"1. **α-螺旋 (Alpha Helix)**\n",
" - **描述**：一个右手螺旋结构，每个氨基酸残基沿螺旋轴旋转约 100 度，并沿着轴向上移动约 1.5 Å。\n",
" - **特点**：通过相邻的肽键之间形成的氢键稳定，通常每 3.6 个氨基酸残基转一圈。\n",
"\n",
"2. **β-折叠片 (Beta Sheet)**\n",
" - **描述**：由多个几乎平行或反平行排列的多肽链组成，链间通过氢键连接。\n",
" - **特点**：可以是平行（所有链同向）或反平行（相邻链方向相反），提供了高度刚性的平面结构。\n",
"\n",
"3. **转角 (Turns)**\n",
" - **描述**：短的序列片段，通常包含 3 到 4 个氨基酸残基，用于改变多肽链的方向。\n",
" - **特点**：最常见的类型是 β-转角（beta turn），它使得链可以在空间上回折。\n",
"\n",
"4. **无规则卷曲 (Random Coil)**\n",
" - **描述**：没有固定模式的区域，可能是由于缺乏足够的氢键或其他稳定力。\n",
" - **特点**：虽然称为“无规则”，但实际上可能在特定环境下具有功能性意义。\n",
"\n",
"\n",
"<img src=\"img/protein-structure-1.png\" width=\"500px\" />\n",
"\n",
"蛋白质的二级结构经常用图形来形象的描述。比如下图中黄色的箭头代表对应的氨基酸 具有β折片结构。波浪线代表螺旋结构，小鼓包是转角。此外，以字母形式书写的二级结构序列能够更加精准的描述。\n",
"其中，E 代表β折叠，H 代表α螺旋，T 代表转角。没有写任何字母的地方是松散的 coil 结构。很多序列预测数据集中，一般不区分转角和coil结构。\n",
"\n",
"\n",
"<img src=\"img/protein-structure-2.png\" width=\"500px\" />\n"
]
},
{
"cell_type": "markdown",
"id": "c90a583c-f6a5-4a41-8e7e-da27b7e95c50",
"metadata": {},
"source": [
"获得实验测定的蛋白质或 RNA 的二级结构数据，通常需要依赖于实验室技术和公共数据库中已发表的实验结果。以下是一些常用的资源和方法，帮助你获取经过实验验证的二级结构数据：\n",
"\n",
"### 1. **蛋白质二级结构数据**\n",
"\n",
"#### a. **PDB (Protein Data Bank)**\n",
"\n",
"- **网址**：[RCSB PDB](https://www.rcsb.org/)\n",
"- **特点**：PDB 是一个全球性的生物大分子结构数据库，包含通过 X 射线晶体学、核磁共振（NMR）和冷冻电镜（Cryo-EM）等实验方法测定的蛋白质三维结构。\n",
"- **使用方法**：\n",
" - 搜索特定蛋白质的 PDB ID 或名称。\n",
" - 查看详细条目页面，其中包含了蛋白质的三级结构信息，可以通过可视化工具如 PyMOL 或 Chimera 来观察二级结构元素（如 α-螺旋、β-折叠片等）。\n",
"\n",
"<img src=\"img/pdb1.png\" width=\"600px\" />\n",
"\n",
"From https://www.rcsb.org/sequence/9rsa\n",
"\n",
"#### b. **PDBe (Protein Data Bank in Europe)**\n",
"\n",
"- **网址**：[PDBe](https://www.ebi.ac.uk/pdbe/)\n",
"- **特点**：PDBe 是欧洲的 PDB 镜像站点，提供了与 RCSB PDB 类似的功能，并且有额外的分析工具和注释信息。\n",
"- **使用方法**：\n",
" - 搜索蛋白质的 PDB ID 或名称。\n",
" - 使用 PDBe-KB 和其他工具来获取详细的结构信息和二级结构注释。\n",
"\n",
"#### c. **Biomolecule Structure Knowledgebase (BSK)**\n",
"\n",
"- **网址**：[BSK](https://bsk.pdbj.org/)\n",
"- **特点**：BSK 是日本的 PDB 镜像站点，同样提供丰富的结构数据和分析工具。\n",
"- **使用方法**：\n",
" - 搜索蛋白质的 PDB ID 或名称。\n",
" - 浏览条目以获取详细的结构信息和二级结构注释。\n",
"\n",
"\n",
"\n",
"### 3. **实验方法**\n",
"\n",
"如果你需要最新的或特定条件下的二级结构数据，可能需要参考文献中的实验方法。以下是一些常见的实验技术：\n",
"\n",
"#### a. **X 射线晶体学**\n",
"\n",
"- **原理**：通过解析蛋白质或 RNA 晶体的衍射图案来确定其三维结构。\n",
"- **应用**：适用于能够形成稳定晶体的分子。\n",
"\n",
"#### b. **核磁共振（NMR）**\n",
"\n",
"- **原理**：利用核磁共振波谱技术来确定溶液状态下分子的结构。\n",
"- **应用**：适用于较小的蛋白质和 RNA 分子。\n",
"\n",
"#### c. **冷冻电镜（Cryo-EM）**\n",
"\n",
"- **原理**：通过低温冷冻样品并在电子显微镜下成像来确定分子结构。\n",
"- **应用**：适用于较大的复合物和难以结晶的分子。\n",
"\n",
"\n",
"\n",
"### 4. **文献检索**\n",
"\n",
"#### a. **PubMed**\n",
"\n",
"- **网址**：[PubMed](https://pubmed.ncbi.nlm.nih.gov/)\n",
"- **特点**：PubMed 是一个广泛使用的生物医学文献数据库，提供了大量关于蛋白质和 RNA 功能及结构的研究论文。\n",
"- **使用方法**：\n",
" - 使用关键词搜索与特定蛋白质或 RNA 相关的实验研究。\n",
" - 阅读论文以获取详细的实验数据和二级结构描述。\n",
"\n",
"### 总结\n",
"\n",
"获得实验测定的蛋白质或 RNA 的二级结构数据主要依赖于公共数据库如 PDB 和 NDB，这些数据库收录了通过多种实验方法测定的结构信息。此外，查阅相关文献也是一种重要的途径，可以找到最新的或特定条件下的实验结果。对于具体的实验方法，如 X 射线晶体学、NMR 和 Cryo-EM 等，它们各自有适用的场景和优势。\n"
]
},
{
"cell_type": "markdown",
"id": "1cadfd11-2130-429d-848f-39371356ca10",
"metadata": {},
"source": [
"## 整理好的数据\n",
"\n",
"https://huggingface.co/datasets/proteinea/secondary_structure_prediction\n",
"\n",
"<img src=\"img/ds_structure.png\" width=\"600px\" />\n",
"\n",
"https://huggingface.co/datasets/genbio-ai/rna-secondary-structure-prediction"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "134a72e3-597a-446e-9193-d060a6e677f6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "b43dd5f2-6b23-4b51-ad04-7b7ded732cb7",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoModel, TrainingArguments, Trainer\n",
"from tokenizers import Tokenizer\n",
"from transformers import GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
"from transformers import AutoModelForTokenClassification \n",
"from transformers import DataCollatorWithPadding"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "4c66fa5b-b8b8-4dfd-ada1-32ed9e690c33",
"metadata": {},
"outputs": [],
"source": [
"#set tokenizer,dna protein \n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"dnagpt/gene_eng_gpt2_v0\")\n",
"tokenizer.pad_token = tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "70a3fd79-48bf-4452-a7ee-689f1b11e987",
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"# 1. load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"proteinea/secondary_structure_prediction\")['train'].train_test_split(test_size=0.1)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "13cd141e-98c3-47da-8e21-cba5576707fe",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['input', 'dssp3', 'dssp8', 'disorder', 'cb513_mask'],\n",
" num_rows: 9712\n",
" })\n",
" test: Dataset({\n",
" features: ['input', 'dssp3', 'dssp8', 'disorder', 'cb513_mask'],\n",
" num_rows: 1080\n",
" })\n",
"})"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "7936af74-3f5f-43c1-aa69-fd7b08989e24",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'input': 'MTQTQPVTPTPPASFQTQHDPRTRLGATPLPGGAGTRFRLWTSTARTVAVRVNGTEHVMTSLGGGIYELELPVGPGARYLFVLDGVPTPDPYARFLPDGVHGEAEVVDFGTFDWTDADWHGIKLADCVFYEVHVGTFTPEGTYRAAAEKLPYLKELGVTAIQVMPLAAFDGQRGWGYDGAAFYAPYAPYGRPEDLMALVDAAHRLGLGVFLDVVYNHFGPSGNYLSSYAPSYFTDRFSSAWGMGLDYAEPHMRRYVTGNARMWLRDYHFDGLRLDATPYMTDDSETHILTELAQEIHELGGTHLLLAEDHRNLPDLVTVNHLDGIWTDDFHHETRVTLTGEQEGYYAGYRGGAEALAYTIRRGWRYEGQFWAVKGEEHERGHPSDALEAPNFVYCIQNHDQIGNRPLGERLHQSDGVTLHEYRGAAALLLTLPMTPLLFQGQEWAASTPFQFFSDHAGELGQAVSEGRKKEFGGFSGFSGEDVPDPQAEQTFLNSKLNWAEREGGEHARTLRLYRDLLRLRREDPVLHNRQRENLTTGHDGDVLWVRTVTGAGERVLLWNLGQDTRAVAEVKLPFTVPRRLLLHTEGREDLTLGAGEAVLVG',\n",
" 'dssp3': 'CCCCCCCCCCCCCCCCCCCCHHHCCEEEECHHHCCEEEEEECCCCCCEEEEECCEEEECEEEECCEEEEEECCCCCCEEEEEECCEEECCCCCCCCCCCCCCCEECCCCCCCCCCCCCCCCCCHHHCCEEEECHHHHCCCCCHHHHHHCHHHHHHHCCCEEEECCCEECCCCCCCCCCCCEEEEECHHHCCHHHHHHHHHHHHHCCCEEEEEECCCCCCCCCCCHHHHCHHHEEEEEECCCCEEECCCCHHHHHHHHHHHHHHHHHHCCCEEEECCHHHCCCCCCCCHHHHHHHHHHCCCCCCEEEEECCCCCCHHHHCCCCCEEECCHHHHHHHHHHHCCCCHHHHHCCCCHHHHHHHHHHCCCCEEEEECCCCCCEEEECCCCCCCHHHEEEECCCHHHHHCCCCCCCHHHCCCCCHHHHHHHHHHHHHCCCEEEEECCHHHCCCCCCCCCCCCCHHHHHHHHHHHHHHCCCCCCCCCCCCCCCCCHHHHHCCCCCCHHHHCHHHHHHHHHHHHHHHHHHHCCCCCCCCHHHEEEEEECCEEEEEEEECCEEEEEEEECCCCCEEHHHCCCCCCCCCCEEEECCCCCCCEECCCCEEEEC',\n",
" 'dssp8': 'CCCCCCCCCCCCCCCCCSCCGGGCSEEEECGGGCCEEEEEECSSCSSEEEEETTEEEECEEEETTEEEEEESCCTTCEEEEEETTEEECCTTCSCCTTCTTSCEECCCTTSSCCCCTTCCCCCGGGCCEEEECHHHHSSSCSHHHHHHTHHHHHHHTCCEEEECCCEECSSSCCCSTTCCEEEEECGGGCCHHHHHHHHHHHHHTTCEEEEEECCSCCCSSSCCHHHHCGGGEEEEEECSSSEEECTTSHHHHHHHHHHHHIIIIIHCCSEEEETTGGGCCCCSSSCHHHHHHHHHHTTCSCCEEEEECSSCCTHHHHTTCCSEEECTHHHHHHHHHHHCCCSGGGGGCCCSHHHHHHHHHHSSSCEEEEECCTTCCEEEECCCTTCCGGGEEEESCCHHHHHTSTTCCCGGGSTTCCHHHHHHHHHHHHHSSSEEEEETTGGGTCSSCCCCCCCCCHHHHHHHHHHHHHHCCCCCCCCCCCCCCTTSHHHHHTTSCCSGGGGSHHHHHHHHHHHHHHHHHHHCTTTTCCCGGGEEEEEETTEEEEEEEETTEEEEEEEECSSSCEEGGGSCCSSCCCCCEEEETTCCSSSEECTTCEEEEC',\n",
" 'disorder': '0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0',\n",
" 'cb513_mask': '1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0'}"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset[\"train\"][0]"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "47b1ac0c-e934-4ac3-b869-509515b15aa1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dna datasets mean token lenght 96.07685185185186 min token length 7 max token length 576\n"
]
}
],
"source": [
"token_len_list = []\n",
"for item in dataset[\"test\"]:\n",
" inputs = tokenizer.tokenize(item[\"input\"])\n",
" token_len_list.append( len(inputs) )\n",
"\n",
"mean_len = sum(token_len_list)/len(token_len_list)\n",
"min_len = min(token_len_list)\n",
"max_len = max(token_len_list)\n",
"\n",
"print(\"dna datasets \", \"mean token lenght\", mean_len, \"min token length\", min_len, \"max token length\", max_len)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "1b32de6e-fe08-426e-983e-7dd157c9af62",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Number of unique labels: 3\n",
"Label to ID mapping: {'C': 0, 'H': 1, 'E': 2, '<pad>': 3}\n"
]
}
],
"source": [
"from collections import Counter\n",
"\n",
"# Confirm the number of labels and create a mapping from string labels to integer IDs.\n",
"all_labels = [label for item in dataset[\"train\"] for label in item[\"dssp3\"]]\n",
"label_counts = Counter(all_labels)\n",
"num_labels = len(label_counts)\n",
"\n",
"# Define a special ID for padding. Make sure this ID is not used by any actual label.\n",
"# If you have 3 classes, start with 3 or higher.\n",
"pad_token_label_id = num_labels # Assuming no other labels have this ID.\n",
"\n",
"label_to_id = {label: i for i, (label, _) in enumerate(label_counts.items())}\n",
"label_to_id['<pad>'] = pad_token_label_id # Add padding token to the mapping.\n",
"id_to_label = {v: k for k, v in label_to_id.items()}\n",
"\n",
"print(f\"Number of unique labels: {num_labels}\")\n",
"print(\"Label to ID mapping:\", label_to_id)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"id": "2bd65f47-3325-4357-a896-9a0abf160e8a",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2ForTokenClassification were not initialized from the model checkpoint at dnagpt/gene_eng_gpt2_v0 and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"#set model\n",
"#model = AutoModelForTokenClassification.from_pretrained('dnagpt/gene_eng_gpt2_v0', )\n",
"model = AutoModelForTokenClassification.from_pretrained(\n",
" 'dnagpt/gene_eng_gpt2_v0',\n",
" num_labels=num_labels + 1, # Include the padding label in the count.\n",
" id2label=id_to_label,\n",
" label2id=label_to_id\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "e247ac1e-bcd4-4aaf-9f91-dc939e5abe89",
"metadata": {},
"outputs": [],
"source": [
"# 5. Preprocess the data\n",
"from transformers import DataCollatorForTokenClassification\n",
"import torch\n",
"# Define the maximum sequence length based on your model or dataset requirements.\n",
"max_seq_length = 128 # Adjust this value as needed.\n",
"\n",
"def preprocess_function(examples):\n",
" tokenized_inputs = tokenizer(\n",
" examples[\"input\"], \n",
" truncation=True, \n",
" padding='max_length', \n",
" max_length=max_seq_length,\n",
" return_tensors=\"pt\" # Return PyTorch tensors directly.\n",
" )\n",
" \n",
" labels = []\n",
" for label in examples['dssp3']:\n",
" label_ids = [label_to_id[l] if l in label_to_id else pad_token_label_id for l in label]\n",
" # Ensure labels are padded/truncated to the same length as inputs.\n",
" if len(label_ids) > max_seq_length:\n",
" label_ids = label_ids[:max_seq_length]\n",
" else:\n",
" label_ids = label_ids + [pad_token_label_id] * (max_seq_length - len(label_ids))\n",
" \n",
" labels.append(label_ids)\n",
" \n",
" tokenized_inputs[\"labels\"] = torch.tensor(labels)\n",
"\n",
" return tokenized_inputs"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "8144d093-e8d3-41ff-ae4f-82aa1f28d689",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "707978d4f8304cada1041f8e794d79b7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/9712 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7ab7ae3ed05244bab1fe13050aad3764",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/1080 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tokenized_datasets = dataset.map(preprocess_function, batched=True)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "de5067da-a010-4e0d-b99b-659ee2d3cf3c",
"metadata": {},
"outputs": [],
"source": [
"# Remove columns that are not required by the model.\n",
"columns_to_remove = ['input', 'dssp3', 'dssp8', 'disorder', 'cb513_mask']\n",
"tokenized_datasets.set_format(\"torch\", columns=[\"input_ids\", \"attention_mask\", \"labels\"], output_all_columns=True)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "fcce6f4e-9716-4fe4-9250-e201a442bbbc",
"metadata": {},
"outputs": [],
"source": [
"# Set up data collator for handling padding during batching.\n",
"data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer, pad_to_multiple_of=8, label_pad_token_id=pad_token_label_id)"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "fa3e62b3-dba4-4cef-9bb7-de410f4bb444",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/transformers/training_args.py:1575: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
" warnings.warn(\n",
"/tmp/ipykernel_1443/204012889.py:41: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n",
" trainer = Trainer(\n"
]
}
],
"source": [
"# 6. Prepare training\n",
"import evaluate\n",
"import numpy as np\n",
"\n",
"metric = evaluate.load(\"seqeval\")\n",
"\n",
"def compute_metrics(p):\n",
" predictions, labels = p\n",
" predictions = np.argmax(predictions, axis=2)\n",
"\n",
" # Remove ignored index (special tokens)\n",
" true_predictions = [\n",
" [id_to_label[p] for (p, l) in zip(prediction, label) if l != pad_token_label_id]\n",
" for prediction, label in zip(predictions, labels)\n",
" ]\n",
" true_labels = [\n",
" [id_to_label[l] for (p, l) in zip(prediction, label) if l != pad_token_label_id]\n",
" for prediction, label in zip(predictions, labels)\n",
" ]\n",
"\n",
" results = metric.compute(predictions=true_predictions, references=true_labels)\n",
" return {\n",
" \"precision\": results[\"overall_precision\"],\n",
" \"recall\": results[\"overall_recall\"],\n",
" \"f1\": results[\"overall_f1\"],\n",
" \"accuracy\": results[\"overall_accuracy\"],\n",
" }\n",
" \n",
"training_args = TrainingArguments(\n",
" output_dir=\"./results\",\n",
" evaluation_strategy=\"epoch\",\n",
" learning_rate=1e-5,\n",
" lr_scheduler_type=\"constant_with_warmup\",\n",
" optim='adamw_torch',\n",
" per_device_train_batch_size=16,\n",
" per_device_eval_batch_size=16,\n",
" num_train_epochs=20,\n",
" weight_decay=0.01,\n",
")\n",
"\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"test\"],\n",
" tokenizer=tokenizer,\n",
" data_collator=DataCollatorWithPadding(tokenizer=tokenizer),\n",
" compute_metrics=compute_metrics,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 38,
"id": "8a76f326-1097-47bb-bb9d-03b77c4f8f4f",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='8001' max='12140' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [ 8001/12140 09:41 < 05:00, 13.76 it/s, Epoch 13.18/20]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" <th>Precision</th>\n",
" <th>Recall</th>\n",
" <th>F1</th>\n",
" <th>Accuracy</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>1.102200</td>\n",
" <td>0.923186</td>\n",
" <td>0.314843</td>\n",
" <td>0.125521</td>\n",
" <td>0.179485</td>\n",
" <td>0.503214</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.942100</td>\n",
" <td>0.883362</td>\n",
" <td>0.357413</td>\n",
" <td>0.153466</td>\n",
" <td>0.214731</td>\n",
" <td>0.521366</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.898500</td>\n",
" <td>0.895442</td>\n",
" <td>0.355443</td>\n",
" <td>0.194234</td>\n",
" <td>0.251199</td>\n",
" <td>0.522545</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>0.870200</td>\n",
" <td>0.891230</td>\n",
" <td>0.367170</td>\n",
" <td>0.050731</td>\n",
" <td>0.089145</td>\n",
" <td>0.526761</td>\n",
" </tr>\n",
" <tr>\n",
" <td>5</td>\n",
" <td>0.831900</td>\n",
" <td>0.890030</td>\n",
" <td>0.373252</td>\n",
" <td>0.197096</td>\n",
" <td>0.257971</td>\n",
" <td>0.530358</td>\n",
" </tr>\n",
" <tr>\n",
" <td>6</td>\n",
" <td>0.815800</td>\n",
" <td>0.867876</td>\n",
" <td>0.378696</td>\n",
" <td>0.236628</td>\n",
" <td>0.291262</td>\n",
" <td>0.540153</td>\n",
" </tr>\n",
" <tr>\n",
" <td>7</td>\n",
" <td>0.800900</td>\n",
" <td>0.873521</td>\n",
" <td>0.380925</td>\n",
" <td>0.212640</td>\n",
" <td>0.272927</td>\n",
" <td>0.544393</td>\n",
" </tr>\n",
" <tr>\n",
" <td>8</td>\n",
" <td>0.785100</td>\n",
" <td>0.872138</td>\n",
" <td>0.385372</td>\n",
" <td>0.156363</td>\n",
" <td>0.222462</td>\n",
" <td>0.547684</td>\n",
" </tr>\n",
" <tr>\n",
" <td>9</td>\n",
" <td>0.774100</td>\n",
" <td>0.885855</td>\n",
" <td>0.384813</td>\n",
" <td>0.180280</td>\n",
" <td>0.245531</td>\n",
" <td>0.549681</td>\n",
" </tr>\n",
" <tr>\n",
" <td>10</td>\n",
" <td>0.750800</td>\n",
" <td>0.884582</td>\n",
" <td>0.388464</td>\n",
" <td>0.206529</td>\n",
" <td>0.269681</td>\n",
" <td>0.555933</td>\n",
" </tr>\n",
" <tr>\n",
" <td>11</td>\n",
" <td>0.737500</td>\n",
" <td>0.886323</td>\n",
" <td>0.396929</td>\n",
" <td>0.202713</td>\n",
" <td>0.268369</td>\n",
" <td>0.557624</td>\n",
" </tr>\n",
" <tr>\n",
" <td>12</td>\n",
" <td>0.731000</td>\n",
" <td>0.878285</td>\n",
" <td>0.365956</td>\n",
" <td>0.315728</td>\n",
" <td>0.338991</td>\n",
" <td>0.555857</td>\n",
" </tr>\n",
" <tr>\n",
" <td>13</td>\n",
" <td>0.708900</td>\n",
" <td>0.912278</td>\n",
" <td>0.377030</td>\n",
" <td>0.249346</td>\n",
" <td>0.300174</td>\n",
" <td>0.555030</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: C seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: H seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/sequence_labeling.py:171: UserWarning: <pad> seems not to be NE tag.\n",
" warnings.warn('{} seems not to be NE tag.'.format(chunk))\n",
"/root/miniconda3/lib/python3.12/site-packages/seqeval/metrics/v1.py:57: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.\n",
" _warn_prf(average, modifier, msg_start, len(result))\n"
]
},
{
"ename": "RuntimeError",
"evalue": "[enforce fail at inline_container.cc:595] . unexpected pos 1216226560 vs 1216226452",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/torch/serialization.py:628\u001b[0m, in \u001b[0;36msave\u001b[0;34m(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization, _disable_byteorder_record)\u001b[0m\n\u001b[1;32m 627\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _open_zipfile_writer(f) \u001b[38;5;28;01mas\u001b[39;00m opened_zipfile:\n\u001b[0;32m--> 628\u001b[0m \u001b[43m_save\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mopened_zipfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpickle_module\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpickle_protocol\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_disable_byteorder_record\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 629\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/torch/serialization.py:862\u001b[0m, in \u001b[0;36m_save\u001b[0;34m(obj, zip_file, pickle_module, pickle_protocol, _disable_byteorder_record)\u001b[0m\n\u001b[1;32m 861\u001b[0m num_bytes \u001b[38;5;241m=\u001b[39m storage\u001b[38;5;241m.\u001b[39mnbytes()\n\u001b[0;32m--> 862\u001b[0m \u001b[43mzip_file\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mwrite_record\u001b[49m\u001b[43m(\u001b[49m\u001b[43mname\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstorage\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_bytes\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mRuntimeError\u001b[0m: [enforce fail at inline_container.cc:764] . PytorchStreamWriter failed writing file data/94: file write failed",
"\nDuring handling of the above exception, another exception occurred:\n",
"\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[38], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# Start training\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[43mtrainer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/transformers/trainer.py:2164\u001b[0m, in \u001b[0;36mTrainer.train\u001b[0;34m(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)\u001b[0m\n\u001b[1;32m 2162\u001b[0m hf_hub_utils\u001b[38;5;241m.\u001b[39menable_progress_bars()\n\u001b[1;32m 2163\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 2164\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43minner_training_loop\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2165\u001b[0m \u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2166\u001b[0m \u001b[43m \u001b[49m\u001b[43mresume_from_checkpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mresume_from_checkpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2167\u001b[0m \u001b[43m \u001b[49m\u001b[43mtrial\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtrial\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2168\u001b[0m \u001b[43m \u001b[49m\u001b[43mignore_keys_for_eval\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mignore_keys_for_eval\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 2169\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/transformers/trainer.py:2591\u001b[0m, in \u001b[0;36mTrainer._inner_training_loop\u001b[0;34m(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)\u001b[0m\n\u001b[1;32m 2589\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstate\u001b[38;5;241m.\u001b[39mepoch \u001b[38;5;241m=\u001b[39m epoch \u001b[38;5;241m+\u001b[39m (step \u001b[38;5;241m+\u001b[39m \u001b[38;5;241m1\u001b[39m \u001b[38;5;241m+\u001b[39m steps_skipped) \u001b[38;5;241m/\u001b[39m steps_in_epoch\n\u001b[1;32m 2590\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_handler\u001b[38;5;241m.\u001b[39mon_step_end(args, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstate, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol)\n\u001b[0;32m-> 2591\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_maybe_log_save_evaluate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2592\u001b[0m \u001b[43m \u001b[49m\u001b[43mtr_loss\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgrad_norm\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtrial\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepoch\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mignore_keys_for_eval\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstart_time\u001b[49m\n\u001b[1;32m 2593\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 2594\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 2595\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_handler\u001b[38;5;241m.\u001b[39mon_substep_end(args, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstate, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol)\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/transformers/trainer.py:3056\u001b[0m, in \u001b[0;36mTrainer._maybe_log_save_evaluate\u001b[0;34m(self, tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval, start_time)\u001b[0m\n\u001b[1;32m 3053\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol\u001b[38;5;241m.\u001b[39mshould_save \u001b[38;5;241m=\u001b[39m is_new_best_metric\n\u001b[1;32m 3055\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol\u001b[38;5;241m.\u001b[39mshould_save:\n\u001b[0;32m-> 3056\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_save_checkpoint\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtrial\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3057\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcallback_handler\u001b[38;5;241m.\u001b[39mon_save(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39margs, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstate, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrol)\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/transformers/trainer.py:3192\u001b[0m, in \u001b[0;36mTrainer._save_checkpoint\u001b[0;34m(self, model, trial)\u001b[0m\n\u001b[1;32m 3188\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msave_model(output_dir, _internal_call\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m 3190\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39margs\u001b[38;5;241m.\u001b[39msave_only_model:\n\u001b[1;32m 3191\u001b[0m \u001b[38;5;66;03m# Save optimizer and scheduler\u001b[39;00m\n\u001b[0;32m-> 3192\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_save_optimizer_and_scheduler\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutput_dir\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3193\u001b[0m \u001b[38;5;66;03m# Save RNG state\u001b[39;00m\n\u001b[1;32m 3194\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_save_rng_state(output_dir)\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/transformers/trainer.py:3313\u001b[0m, in \u001b[0;36mTrainer._save_optimizer_and_scheduler\u001b[0;34m(self, output_dir)\u001b[0m\n\u001b[1;32m 3308\u001b[0m save_fsdp_optimizer(\n\u001b[1;32m 3309\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maccelerator\u001b[38;5;241m.\u001b[39mstate\u001b[38;5;241m.\u001b[39mfsdp_plugin, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39maccelerator, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39moptimizer, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel, output_dir\n\u001b[1;32m 3310\u001b[0m )\n\u001b[1;32m 3311\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39margs\u001b[38;5;241m.\u001b[39mshould_save:\n\u001b[1;32m 3312\u001b[0m \u001b[38;5;66;03m# deepspeed.save_checkpoint above saves model/optim/sched\u001b[39;00m\n\u001b[0;32m-> 3313\u001b[0m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msave\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43moptimizer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstate_dict\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mos\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpath\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mjoin\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutput_dir\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mOPTIMIZER_NAME\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3315\u001b[0m \u001b[38;5;66;03m# Save SCHEDULER & SCALER\u001b[39;00m\n\u001b[1;32m 3316\u001b[0m is_deepspeed_custom_scheduler \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mis_deepspeed_enabled \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\n\u001b[1;32m 3317\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlr_scheduler, DeepSpeedSchedulerWrapper\n\u001b[1;32m 3318\u001b[0m )\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/torch/serialization.py:627\u001b[0m, in \u001b[0;36msave\u001b[0;34m(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization, _disable_byteorder_record)\u001b[0m\n\u001b[1;32m 624\u001b[0m _check_save_filelike(f)\n\u001b[1;32m 626\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m _use_new_zipfile_serialization:\n\u001b[0;32m--> 627\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43;01mwith\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43m_open_zipfile_writer\u001b[49m\u001b[43m(\u001b[49m\u001b[43mf\u001b[49m\u001b[43m)\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mas\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mopened_zipfile\u001b[49m\u001b[43m:\u001b[49m\n\u001b[1;32m 628\u001b[0m \u001b[43m \u001b[49m\u001b[43m_save\u001b[49m\u001b[43m(\u001b[49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mopened_zipfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpickle_module\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpickle_protocol\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m_disable_byteorder_record\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 629\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43;01mreturn\u001b[39;49;00m\n",
"File \u001b[0;32m~/miniconda3/lib/python3.12/site-packages/torch/serialization.py:475\u001b[0m, in \u001b[0;36m_open_zipfile_writer_file.__exit__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 474\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__exit__\u001b[39m(\u001b[38;5;28mself\u001b[39m, \u001b[38;5;241m*\u001b[39margs) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 475\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfile_like\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mwrite_end_of_file\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 476\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfile_stream \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 477\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfile_stream\u001b[38;5;241m.\u001b[39mclose()\n",
"\u001b[0;31mRuntimeError\u001b[0m: [enforce fail at inline_container.cc:595] . unexpected pos 1216226560 vs 1216226452"
]
}
],
"source": [
"# Start training\n",
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "950c460f-5631-4c9a-819b-1e3ac484cc65",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'eval_loss': 0.9122781157493591,\n",
" 'eval_precision': 0.3770299145299145,\n",
" 'eval_recall': 0.2493464283190843,\n",
" 'eval_f1': 0.3001743716242079,\n",
" 'eval_accuracy': 0.5550300748427384}"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results = trainer.evaluate()\n",
"results"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "8174c1c6-a5bc-4fe3-8f9b-356625531e7d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
">>> Perplexity: 2.49\n"
]
}
],
"source": [
"import math\n",
"eval_results = trainer.evaluate()\n",
"print(f\">>> Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
]
},
{
"cell_type": "code",
"execution_count": 41,
"id": "6a22f131-9e5f-4125-942a-22d1b1e6373b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('./secondary_structure_model/tokenizer_config.json',\n",
" './secondary_structure_model/special_tokens_map.json',\n",
" './secondary_structure_model/vocab.json',\n",
" './secondary_structure_model/merges.txt',\n",
" './secondary_structure_model/added_tokens.json')"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 保存模型\n",
"model.save_pretrained(\"./secondary_structure_model\")\n",
"tokenizer.save_pretrained(\"./secondary_structure_model\")"
]
},
{
"cell_type": "code",
"execution_count": 42,
"id": "d5817a6c-c707-4005-9210-2a12ff0d43b0",
"metadata": {},
"outputs": [],
"source": [
"# 加载模型\n",
"model = AutoModelForTokenClassification.from_pretrained(\"./secondary_structure_model\")\n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"./secondary_structure_model\")"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "2f6ebdc6-8ff8-4947-ada4-05ff4b28e0f3",
"metadata": {},
"outputs": [],
"source": [
"# 进行预测\n",
"def predict_secondary_structure(sequence):\n",
" inputs = tokenizer(sequence, return_tensors=\"pt\", truncation=True, padding=True)\n",
" outputs = model(**inputs)\n",
" predictions = outputs.logits.argmax(dim=-1)\n",
" return predictions"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "841ebba8-7619-411f-a11e-841de3a3f064",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[0, 0, 0, 0, 2, 2, 2, 2]])\n"
]
}
],
"source": [
"# 示例预测\n",
"sequence = \"ACDEFGHIKLMNPQRSTVWY\"\n",
"predictions = predict_secondary_structure(sequence)\n",
"print(predictions)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "37e7d22e-0545-422b-b8ba-7990ca127d8a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03-gene-task/3-multi-seq-task.ipynb
+865
View File
File diff suppressed because one or more lines are too long
03-gene-task/4-fun-predict.ipynb
+787
View File
@@ -0,0 +1,787 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ce0fa061-3f49-46c3-ba5c-8dcca7d283d3",
"metadata": {},
"source": [
"# 3.4 功能预测任务"
]
},
{
"cell_type": "markdown",
"id": "e0fe3429-dbff-4e83-827a-2e31da60dfc3",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## 数据准备\n",
"\n",
"获得蛋白质序列及其对应的功能描述是生物信息学中的一个重要任务，通常涉及从公共数据库中检索数据或通过实验方法确定。以下是几种常用的方法和资源，帮助你获取蛋白质序列和功能描述：\n",
"\n",
"### 1. **使用公共数据库**\n",
"\n",
"#### a. **UniProt (Universal Protein Resource)**\n",
"\n",
"- **网址**：[UniProt](https://www.uniprot.org/)\n",
"- **特点**：UniProt 是一个综合性的蛋白质数据库，提供了丰富的注释信息，包括蛋白质序列、结构、功能、亚细胞定位等。\n",
"- **使用方法**：\n",
" - 在搜索栏中输入蛋白质名称、基因名称或序列 ID。\n",
" - 浏览结果页面以查看详细的注释信息，包括功能描述、GO（Gene Ontology）术语、文献引用等。\n",
"\n",
"\n",
"<img src=\"img/function.png\" width=\"500px\" />\n",
"\n",
"<img src=\"img/sequence.png\" width=\"500px\" />\n",
"\n",
"#### b. **NCBI (National Center for Biotechnology Information)**\n",
"\n",
"- **网址**：[NCBI](https://www.ncbi.nlm.nih.gov/)\n",
"- **特点**：NCBI 提供多个相关数据库，如 GenBank、RefSeq 和 Protein 数据库，涵盖广泛的生物物种和蛋白质信息。\n",
"- **使用方法**：\n",
" - 使用 NCBI 的搜索工具 Entrez 或 BLAST 搜索蛋白质序列或功能描述。\n",
" - 访问特定的蛋白质条目以获取详细信息，包括序列、功能、参考文献等。\n",
"\n",
"#### c. **PDB (Protein Data Bank)**\n",
"\n",
"- **网址**：[PDB](https://www.rcsb.org/)\n",
"- **特点**：PDB 主要包含蛋白质的三维结构信息，但也提供相关的功能描述和文献引用。\n",
"- **使用方法**：\n",
" - 使用 PDB 的搜索功能查找特定蛋白质的结构信息。\n",
" - 查看每个条目的详细页面以获取功能描述和其他相关信息。\n",
"\n",
"#### d. **Ensembl**\n",
"\n",
"- **网址**：[Ensembl](https://www.ensembl.org/)\n",
"- **特点**：Ensembl 提供基因组浏览器和注释信息，涵盖了多种物种的基因和蛋白质数据。\n",
"- **使用方法**：\n",
" - 使用 Ensembl 的搜索功能查找特定蛋白质或基因。\n",
" - 浏览条目页面以获取详细的注释信息，包括功能描述、GO 术语等。\n",
"\n",
"### 2. **通过生物信息学工具**\n",
"\n",
"#### a. **BLAST (Basic Local Alignment Search Tool)**\n",
"\n",
"- **网址**：[BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi)\n",
"- **特点**：BLAST 是一种常用的比对工具，可以帮助你根据已知的蛋白质序列找到相似的序列，并获取其功能描述。\n",
"- **使用方法**：\n",
" - 输入你的蛋白质序列。\n",
" - 选择适当的数据库（如 NR、Swiss-Prot 等）进行比对。\n",
" - 分析比对结果，查看相似序列的功能描述。\n",
"\n",
"#### b. **InterProScan**\n",
"\n",
"- **网址**：[InterPro](https://www.ebi.ac.uk/interpro/)\n",
"- **特点**：InterProScan 是一种用于识别蛋白质家族、结构域和重要位点的工具，可以提供详细的注释信息。\n",
"- **使用方法**：\n",
" - 输入你的蛋白质序列。\n",
" - 运行 InterProScan 分析，获取功能描述、结构域信息等。\n",
"\n",
"### 3. **通过文献和出版物**\n",
"\n",
"#### a. **PubMed**\n",
"\n",
"- **网址**：[PubMed](https://pubmed.ncbi.nlm.nih.gov/)\n",
"- **特点**：PubMed 是一个广泛使用的生物医学文献数据库，提供了大量关于蛋白质功能的研究论文。\n",
"- **使用方法**：\n",
" - 使用关键词搜索与特定蛋白质相关的研究论文。\n",
" - 阅读论文以获取详细的实验数据和功能描述。\n",
"\n",
"#### b. **Google Scholar**\n",
"\n",
"- **网址**：[Google Scholar](https://scholar.google.com/)\n",
"- **特点**：Google Scholar 是一个学术搜索引擎，涵盖广泛的科学文献。\n",
"- **使用方法**：\n",
" - 使用关键词搜索与特定蛋白质相关的研究论文。\n",
" - 阅读论文以获取详细的实验数据和功能描述。"
]
},
{
"cell_type": "markdown",
"id": "e794b698-ef26-4695-9b77-ff7314210e8b",
"metadata": {},
"source": [
"## 整理好的数据\n",
"\n",
"https://huggingface.co/datasets/PharMolix/MutaDescribe\n",
"\n",
"<img src=\"img/dataset.png\" width=\"500px\" />\n",
"\n",
"https://huggingface.co/datasets/jonghyunlee/UniProt_function_text_descriptions?row=2"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "e6c59f74-877a-4a74-9017-e61eb713e285",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f038b5f3-b2a5-45bd-b66a-0475b1f2c026",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"from tokenizers import Tokenizer\n",
"from transformers import GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
"from transformers import AutoModelForSeq2SeqLM \n",
"from transformers import DataCollatorWithPadding"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "7c861666-010e-46d6-aaf0-c63e52920d99",
"metadata": {},
"outputs": [],
"source": [
"#set tokenizer,dna protein \n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"dnagpt/gene_eng_gpt2_v0\")\n",
"tokenizer.pad_token = tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "32532c2d-962d-4aa0-a823-87d6c62a411f",
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"# 1. load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"jonghyunlee/UniProt_function_text_descriptions\")['train'].select(range(5000)).train_test_split(test_size=0.05)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "3829dc86-1274-440b-aa2b-e100f112e9bf",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['entry', 'entry_name', 'protein_name', 'sequence', 'function'],\n",
" num_rows: 4750\n",
" })\n",
" test: Dataset({\n",
" features: ['entry', 'entry_name', 'protein_name', 'sequence', 'function'],\n",
" num_rows: 250\n",
" })\n",
"})"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "6cb4a847-8fd1-4cdd-a905-e0595250c712",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'entry': 'A1TFU9',\n",
" 'entry_name': 'HPXO_MYCVP',\n",
" 'protein_name': 'FAD-dependent urate hydroxylase (EC 1.14.13.113) (Flavoprotein urate hydroxylase)',\n",
" 'sequence': 'MKVVIVGAGMGGMSAAIALRQIGIDTVVYERVTENKPVGAAISVWSNGVKCLNYLGLQEETAELGGKVETMSYVDGHTGDTMCRFSMHPLIEQVGQRPYPIARAELQLMLMKAYGIDDINFGMKMVGVENDTAGSAAKATFADGTTVSADVIIGADGAGSITREYVLGGPVSRRYAGYVNYNGLVSTDDAIGPATEWTTYVGDGKRVSVMPVSDDRFYFFFDVVEPQGSPYEEGRVREVLRAHFAGWTPGVQTLIDTLDPLATNRVEILDLDPFHTWVKGRVAVLGDAAHNTTPDIGQGGCSAMEDAIALQWAFKDHPDDVHAALAAYQSARTERAADLVLRARKRCDVTHAKDPQVTSRWYDELRNEDGTNIIRGIVGNIVGGPLTPVTAATEG',\n",
" 'function': 'Catalyzes the hydroxylation of urate to 5-hydroxyisourate (HIU). Is likely to be involved in the urate degradation pathway to allantoin. Prefers NADH over NADPH as the electron donor. '}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset[\"train\"][0]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "2eb330e7-28ea-46f1-b9bb-093352e1c5d8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dna datasets mean token lenght 269.515 min token length 24 max token length 4577\n"
]
}
],
"source": [
"token_len_list = []\n",
"for item in dataset[\"test\"].select(range(200)):\n",
" inputs = tokenizer.tokenize(item[\"sequence\"])\n",
" token_len_list.append( len(inputs) )\n",
"\n",
"mean_len = sum(token_len_list)/len(token_len_list)\n",
"min_len = min(token_len_list)\n",
"max_len = max(token_len_list)\n",
"\n",
"print(\"dna datasets \", \"mean token lenght\", mean_len, \"min token length\", min_len, \"max token length\", max_len)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "f904a4ad-6fe1-4588-b5fd-5541e26a9bfd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dna datasets mean token lenght 271.02 min token length 23 max token length 1934\n"
]
}
],
"source": [
"token_len_list = []\n",
"for item in dataset[\"test\"].select(range(50)):\n",
" inputs = tokenizer.tokenize(item[\"function\"])\n",
" token_len_list.append( len(inputs) )\n",
"\n",
"mean_len = sum(token_len_list)/len(token_len_list)\n",
"min_len = min(token_len_list)\n",
"max_len = max(token_len_list)\n",
"\n",
"print(\"dna datasets \", \"mean token lenght\", mean_len, \"min token length\", min_len, \"max token length\", max_len)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "73ea434d-6262-4d47-a6e9-3ffb136d8cd0",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "32b5e6b926f24df282da36e7f14cc9ba",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/4750 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6082815e684f4469bee48ada6a049998",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/250 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"max_length = 128\n",
"\n",
"def preprocess_function(examples):\n",
" # 直接从 examples 中提取字段\n",
" inputs = examples[\"sequence\"] # 获取所有样本的 \"sequence\"\n",
" targets = examples[\"function\"] # 获取所有样本的 \"function\"\n",
"\n",
" # 对数据进行编码\n",
" model_inputs = tokenizer(\n",
" inputs, text_target=targets, max_length=max_length, truncation=True\n",
" )\n",
" return model_inputs\n",
"\n",
"\n",
"# 应用分词\n",
"tokenized_datasets = dataset.map(preprocess_function, batched=True, remove_columns=dataset[\"train\"].column_names,)\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "36c17576-cf54-4bfd-8d98-fe41a4b7d2cc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'input_ids': [9206, 609, 532, 2065, 487, 1241, 50, 1785, 4070, 827, 28817, 3840, 1105, 8309, 2993, 3449, 47036, 22588, 5215, 636, 4189, 12265, 3721, 7075, 69183, 3040, 814, 1209, 1910, 217, 474, 13943, 15033, 535, 558, 51164, 12333, 56886, 1174, 338, 20934, 9865, 46, 1131, 3021, 336, 11005, 20318, 748, 396, 46, 38, 46, 54, 12036, 482, 4807, 284, 13333, 87969, 1482, 618, 371, 46, 49, 29703, 46, 5669, 55496, 40, 2682, 2186, 84535, 471, 12020, 280, 1751, 46, 545, 3968, 1660, 354, 1309, 84775, 328, 3802, 52, 46, 33718, 797, 46, 39, 487, 965, 16953, 790, 8503, 53823, 365, 39878, 41235, 17957, 25823, 785, 967, 1371, 543, 8660, 1510, 308, 46, 46, 35663, 3804, 4662, 15100, 8524, 2378, 254, 2399, 38462, 1700, 3223, 1296, 478, 1972, 809, 251], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [31598, 406, 418, 11784, 170, 2001, 170, 4596, 26168, 170, 36668, 69, 170, 3225, 283, 829, 170, 223, 362, 170, 5700, 83, 170, 450, 170, 65, 170, 14420, 270, 170, 2001, 170, 5429, 52715, 468, 370, 3550, 46, 33, 170, 37558, 77, 170, 39404, 73, 65, 170, 1389, 170, 5739, 548, 456, 170, 1389, 170, 15898, 456, 170, 666, 170, 2021, 672, 170, 2001, 170, 13336, 1471, 1882, 170, 59356, 307, 170, 8, 518, 46, 13, 73310, 562, 170, 1389, 170, 518, 46, 13, 1681, 65, 9, 14, 26398, 78, 26168, 170, 36668, 69, 170, 4655, 792, 170, 436, 170, 35585, 24270, 952, 170, 248, 170, 4655, 792, 170, 1978, 170, 21544, 13, 35, 80, 39, 170, 21033, 597, 13, 78252, 952, 3550, 46, 33, 170, 37558, 77]}\n",
"{'input_ids': [45, 504, 1187, 659, 46, 5874, 86301, 412, 51, 86301, 3970, 4570, 59926, 3476, 43, 1517, 46, 244, 46, 335, 9187, 1342, 36689, 14000, 542, 40307, 3757, 14421, 412, 3762, 256, 1588, 723, 12505, 36170, 4898, 846, 87891, 3670, 4020, 4651, 12182, 1121, 60975, 264, 1201, 404, 10714, 256, 1396, 4709, 22460, 2538, 254, 1173, 71302, 423, 11201, 1259, 4013, 87933, 23361, 3410, 46, 41032, 407, 1131, 23083, 2151, 333, 4143, 28020, 213, 52, 21197, 46, 58776, 46, 10528, 1424, 19812, 5974, 46, 54], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [50, 18991, 19522, 170, 1978, 170, 4067, 26168, 170, 1389, 170, 11307, 170, 666, 170, 2250, 85871, 170, 2001, 170, 666, 170, 11086, 655, 1836, 170, 1285, 79, 45056, 17704, 170, 298, 36851, 875, 170, 8371, 6081, 170, 1389, 170, 666, 170, 10434, 10228, 69, 170, 8, 70, 6326, 9, 170, 82935, 468, 69, 14, 26398, 1268, 456, 170, 2901, 170, 70249, 362, 6356, 170, 248, 170, 79448, 456, 170, 1389, 170, 3774, 456, 170, 79448, 69, 170, 11086, 655, 597, 170, 1338, 982, 14, 170]}\n"
]
}
],
"source": [
"# 查看处理后的数据，使用正确的拆分（例如 'train'）\n",
"print(tokenized_datasets['train'][0]) # 查看 'train' 数据集中的第一个样本\n",
"print(tokenized_datasets[\"test\"][0])# 查看 'test' 数据集中的第一个样本\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "9aec8d0b-ef4f-43f3-b986-53edfc7a509f",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "086f8f974e99413da19fc5c18030f1c6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model.safetensors: 69%|######9 | 430M/620M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "187fabd0fd8b49419f7fa1ecf3ff6216",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"generation_config.json: 0%| | 0.00/111 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/transformers/training_args.py:1575: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
" warnings.warn(\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[2025-01-05 16:06:01,794] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/compiler_compat/ld: cannot find -laio: No such file or directory\n",
"collect2: error: ld returned 1 exit status\n",
"/root/miniconda3/compiler_compat/ld: warning: libpthread.so.0, needed by /usr/local/cuda/lib64/libcufile.so, not found (try using -rpath or -rpath-link)\n",
"/root/miniconda3/compiler_compat/ld: warning: libstdc++.so.6, needed by /usr/local/cuda/lib64/libcufile.so, not found (try using -rpath or -rpath-link)\n",
"/root/miniconda3/compiler_compat/ld: warning: libm.so.6, needed by /usr/local/cuda/lib64/libcufile.so, not found (try using -rpath or -rpath-link)\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::runtime_error::~runtime_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__gxx_personality_v0@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream::tellp()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::chrono::_V2::steady_clock::now()@GLIBCXX_3.4.19'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for bool@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_logic_error(char const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::logic_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::locale::~locale()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_end_catch@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_ofstream<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::logic_error::~logic_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for __cxxabiv1::__si_class_type_info@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ios<char, std::char_traits<char> >::_M_cache_locale(std::locale const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `operator new[](unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::_M_leak_hard()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_ifstream<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >::basic_streambuf(std::basic_streambuf<wchar_t, std::char_traits<wchar_t> > const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::append(char const*, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for unsigned short@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::resize(unsigned long, char)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for char const*@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ctype<char>::_M_widen_init() const@GLIBCXX_3.4.11'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_invalid_argument(char const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::locale::operator=(std::locale const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ios<wchar_t, std::char_traits<wchar_t> >::_M_cache_locale(std::locale const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Rb_tree_decrement(std::_Rb_tree_node_base const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_free_exception@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::condition_variable::notify_one()@GLIBCXX_3.4.11'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ios_base::Init::~Init()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_pure_virtual@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream::flush()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for __cxxabiv1::__class_type_info@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_rethrow@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_fstream<char, std::char_traits<char> >::~basic_fstream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::compare(char const*) const@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::locale::locale()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::chrono::_V2::system_clock::now()@GLIBCXX_3.4.19'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_ifstream<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Hash_bytes(void const*, unsigned long, unsigned long)@CXXABI_1.3.5'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<long long>(long long)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for char*@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__detail::_Prime_rehash_policy::_M_need_rehash(unsigned long, unsigned long, unsigned long) const@GLIBCXX_3.4.18'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::out_of_range@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<unsigned long>(unsigned long)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Rb_tree_increment(std::_Rb_tree_node_base const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ios_base::~ios_base()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::range_error::~range_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__basic_file<char>::~__basic_file()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_guard_acquire@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<bool>(bool)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::overflow_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_fstream<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::range_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_ios<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_filebuf<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `operator delete[](void*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(unsigned long, char, std::allocator<char> const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__detail::_List_node_base::_M_transfer(std::__detail::_List_node_base*, std::__detail::_List_node_base*)@GLIBCXX_3.4.15'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::replace(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for std::exception@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_Rep::_M_destroy(std::allocator<wchar_t> const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::istream& std::istream::_M_extract<double>(double&)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_filebuf<char, std::char_traits<char> >::close()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_fstream<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ifstream<char, std::char_traits<char> >::basic_ifstream(char const*, std::_Ios_Openmode)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::append(std::string const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `operator new(unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_istringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for unsigned int@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::append(char const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::domain_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::find(char, unsigned long) const@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream::put(char)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for int@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_bad_alloc()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_thread_atexit@CXXABI_1.3.7'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Rb_tree_increment(std::_Rb_tree_node_base*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ifstream<char, std::char_traits<char> >::~basic_ifstream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ios_base::Init::Init()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::condition_variable::condition_variable()@GLIBCXX_3.4.11'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_filebuf<char, std::char_traits<char> >::basic_filebuf()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::domain_error::~domain_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::cerr@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::find(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::allocator<char> const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::invalid_argument@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for void*@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::assign(std::string const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Rb_tree_rebalance_for_erase(std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for unsigned long@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)@GLIBCXX_3.4.15'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__detail::_List_node_base::_M_unhook()@GLIBCXX_3.4.15'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_ostringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_iostream<char, std::char_traits<char> >::~basic_iostream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::locale::locale(std::locale const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_istringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `log2f@GLIBC_2.2.5'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream::operator<<(std::basic_streambuf<char, std::char_traits<char> >*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::exception::~exception()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__basic_file<char>::is_open() const@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_istringstream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::swap(std::string&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for unsigned long*@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_streambuf<char, std::char_traits<char> >::basic_streambuf(std::basic_streambuf<char, std::char_traits<char> > const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ios<char, std::char_traits<char> >::init(std::basic_streambuf<char, std::char_traits<char> >*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_bad_cast()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ios<char, std::char_traits<char> >::clear(std::_Ios_Iostate)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >::operator=(std::basic_streambuf<wchar_t, std::char_traits<wchar_t> > const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `operator delete(void*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream::operator<<(int)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::_Rep::_S_empty_rep_storage@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::_Rep::_M_destroy(std::allocator<char> const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_iostream<wchar_t, std::char_traits<wchar_t> >::~basic_iostream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::runtime_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_ofstream<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_stringstream()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `VTT for std::basic_stringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<long>(long)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::istream::get()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for unsigned long long@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::out_of_range::~out_of_range()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::length_error::~length_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::invalid_argument::~invalid_argument()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::swap(std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::cout@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<unsigned long long>(unsigned long long)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for int*@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<void const*>(void const*)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::underflow_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_streambuf<char, std::char_traits<char> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for std::out_of_range@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_allocate_exception@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_ios<wchar_t, std::char_traits<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for void const*@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ios<wchar_t, std::char_traits<wchar_t> >::init(std::basic_streambuf<wchar_t, std::char_traits<wchar_t> >*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::reserve(unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_begin_catch@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for long@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_Rep::_S_empty_rep_storage@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::_M_leak()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_filebuf<char, std::char_traits<char> >::open(char const*, std::_Ios_Openmode)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_stringbuf<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::_M_sync(wchar_t*, unsigned long, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::istream::getline(char*, long, char)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_stringstream<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::condition_variable::~condition_variable()@GLIBCXX_3.4.11'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::basic_stringbuf<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::insert(unsigned long, char const*, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::string::assign(char const*, unsigned long)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for unsigned char@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ios_base::ios_base()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_out_of_range(char const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::overflow_error::~overflow_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_length_error(char const*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::__throw_system_error(int)@GLIBCXX_3.4.11'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ofstream<char, std::char_traits<char> >::close()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::ostream& std::ostream::_M_insert<double>(double)@GLIBCXX_3.4.9'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_streambuf<char, std::char_traits<char> >::operator=(std::basic_streambuf<char, std::char_traits<char> > const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `typeinfo for long long@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_ifstream<char, std::char_traits<char> >::close()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_guard_release@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `__cxa_throw@CXXABI_1.3'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::underflow_error::~underflow_error()@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::_Rb_tree_decrement(std::_Rb_tree_node_base*)@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `vtable for std::length_error@GLIBCXX_3.4'\n",
"/root/miniconda3/compiler_compat/ld: /usr/local/cuda/lib64/libcufile.so: undefined reference to `std::basic_filebuf<char, std::char_traits<char> >::~basic_filebuf()@GLIBCXX_3.4'\n",
"collect2: error: ld returned 1 exit status\n"
]
},
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='2970' max='2970' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [2970/2970 03:02, Epoch 5/5]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>5.123600</td>\n",
" <td>4.953184</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>4.958600</td>\n",
" <td>4.907465</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>4.893500</td>\n",
" <td>4.876421</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>4.801000</td>\n",
" <td>4.860046</td>\n",
" </tr>\n",
" <tr>\n",
" <td>5</td>\n",
" <td>4.709600</td>\n",
" <td>4.876781</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=2970, training_loss=4.859284495344066, metrics={'train_runtime': 183.1426, 'train_samples_per_second': 129.68, 'train_steps_per_second': 16.217, 'total_flos': 1551421440000000.0, 'train_loss': 4.859284495344066, 'epoch': 5.0})"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import Trainer, TrainingArguments, GPT2LMHeadModel\n",
"from transformers import DataCollatorForSeq2Seq\n",
"\n",
"\n",
"# 加载预训练的 GPT-2 模型\n",
"model = GPT2LMHeadModel.from_pretrained(\"dnagpt/gene_eng_gpt2_v0\")\n",
"\n",
"data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)\n",
"\n",
"# 训练参数\n",
"training_args = TrainingArguments(\n",
" output_dir=\"gpt2_protein_function\", # 输出目录\n",
" evaluation_strategy=\"epoch\",\n",
" learning_rate=5e-5,\n",
" per_device_train_batch_size=8,\n",
" per_device_eval_batch_size=8,\n",
" num_train_epochs=5,\n",
" save_steps=500,\n",
" save_total_limit=2,\n",
" logging_steps=500,\n",
")\n",
"\n",
"# 初始化 Trainer\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_datasets['train'], # 传递 'train' 拆分数据\n",
" eval_dataset=tokenized_datasets['test'], # 传递 'test' 拆分数据\n",
" data_collator=data_collator,\n",
")\n",
"\n",
"# 训练模型\n",
"trainer.train()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0e9e5526-39ab-4a75-a09d-f4d14a691211",
"metadata": {},
"outputs": [],
"source": [
"# 保存训练好的模型和 tokenizer\n",
"model.save_pretrained(run_path)\n",
"tokenizer.save_pretrained(run_path)\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "18cba087-6620-4d08-ae3a-0ffbd37c7f69",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'eval_loss': 4.8767805099487305, 'eval_runtime': 0.489, 'eval_samples_per_second': 511.252, 'eval_steps_per_second': 65.44, 'epoch': 5.0}\n",
"Perplexity: 131.21\n"
]
}
],
"source": [
"import math\n",
"# 评估模型在验证集上的表现\n",
"eval_results = trainer.evaluate()\n",
"print(eval_results)\n",
"print(f\"Perplexity: {math.exp(eval_results['eval_loss']):.2f}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4e032392-f018-44de-b1f4-0143025a660c",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03-gene-task/5-regression-task.ipynb
+587
View File
@@ -0,0 +1,587 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c499a5c3-0244-41c4-9947-e166206204e2",
"metadata": {},
"source": [
"# 3.5 回归类任务"
]
},
{
"cell_type": "markdown",
"id": "4678171b-bbc8-49dd-ad04-48f5ef89b45e",
"metadata": {},
"source": [
"GPT-2 原本是设计用于生成自然语言的模型，但通过适当的调整和微调，它也可以用于回归任务，例如预测连续值。\n",
"\n",
"使用 GPT-2 进行回归问题的解决，可以将回归问题转化为自回归语言模型任务。GPT-2 原本是设计用于生成自然语言的模型，但通过适当的调整和微调，它也可以用于回归任务，例如预测连续值（如情感评分、价格预测等）。\n",
"\n",
"---\n",
"\n",
"### **1. 使用 GPT-2 做回归的核心思路**\n",
"\n",
"1. **调整输出层**：\n",
" - 默认情况下，GPT-2 的输出是一个词汇表大小的概率分布，用于预测下一个 token。\n",
" - 对于回归问题，可以将模型的最后一层替换为一个线性层，使得输出变为一个标量或多个连续值。\n",
" - gpt2的huggingface实现中，可以简单设置1个分类的分类header，实现回归预测。\n",
"\n",
"2. **损失函数**：\n",
" - 对于回归问题，使用均方误差（MSE）或均绝对误差（MAE）作为损失函数，而不是分类任务中常用的交叉熵。\n",
"\n",
"3. **输入格式**：\n",
" - 输入数据仍然是文本，可以通过特定的模板形式加入上下文信息。\n",
"\n",
"---\n",
"\n",
"### **2. GPT-2 回归任务的实现步骤**\n",
"\n",
"#### **（1）加载基础模型**\n",
"\n",
"从 Hugging Face Transformers 库加载 GPT-2 模型和分词器，并调整其配置以适应回归任务。\n",
"\n",
"```python\n",
"from transformers import GPT2Tokenizer, GPT2Model, GPT2Config, AutoModelForSequenceClassification\n",
"\n",
"# 加载分词器\n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
"\n",
"# 调整模型配置，num_labels=1 表示回归任务\n",
"config = GPT2Config.from_pretrained(\"gpt2\", num_labels=1)\n",
"\n",
"# 加载模型，增加回归输出\n",
"model = AutoModelForSequenceClassification.from_pretrained(\"gpt2\", config=config)\n",
"```\n",
"\n",
"---\n",
"\n",
"### **3. 课程数据集**\n",
"\n",
"本例程使用了蛋白质稳定性分析的数据集，也就是一个蛋白质序列，对应一个float的数值，做回归预测分析。\n",
"\n",
"**蛋白质稳定性分析**是研究蛋白质在不同条件下保持其结构和功能的能力的过程。蛋白质稳定性是生物化学和生物技术领域的重要课题，影响着蛋白质的折叠、功能执行、以及在应用中的可用性（如工业酶、药物开发等）。\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "1e8c0f86-af78-43e1-8db4-e2a2ea22f815",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"\\nimport os\\n\\n# 设置环境变量, autodl专区 其他idc\\nos.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\n# 打印环境变量以确认设置成功\\nprint(os.environ.get('HF_ENDPOINT'))\\n\""
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value\n",
"\n",
"\"\"\"\n",
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c51a8d69-9a36-47e7-8084-f64e6a72e4f7",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoModel\n",
"from tokenizers import Tokenizer\n",
"from transformers import GPT2LMHeadModel, AutoConfig,GPT2Tokenizer\n",
"from transformers import AutoModelForSequenceClassification\n",
"from transformers import DataCollatorWithPadding"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "a5aeb7c1-2d2a-4f57-ad8c-659613870e59",
"metadata": {},
"outputs": [],
"source": [
"#set tokenizer\n",
"tokenizer = GPT2Tokenizer.from_pretrained(\"dnagpt/gene_eng_gpt2_v0\")\n",
"tokenizer.pad_token = tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ad0c19cd-96a5-463e-8b7d-439646fef429",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at dnagpt/gene_eng_gpt2_v0 and are newly initialized: ['score.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
}
],
"source": [
"#set model\n",
"model = AutoModelForSequenceClassification.from_pretrained('dnagpt/gene_eng_gpt2_v0', num_labels=1)\n",
"model.config.pad_token_id = model.config.eos_token_id"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "8c48cb0a-6142-4afc-823e-08fb33f74222",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['seq_id', 'seq_type', 'seq', 'label'],\n",
" num_rows: 62079\n",
" })\n",
" test: Dataset({\n",
" features: ['seq_id', 'seq_type', 'seq', 'label'],\n",
" num_rows: 6898\n",
" })\n",
"})"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset\n",
"# 1. load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"csv\", data_files=\"data/protein_stab.csv\")['train'].train_test_split(test_size=0.1)\n",
"dataset"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "685dd025-f00a-4869-bc30-9843c77b6d8a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'seq_id': 'train_prot_32672',\n",
" 'seq_type': 'prot',\n",
" 'seq': 'FYRLIIFKYPDYIDTYLRLAAIAKEKNNLQLSIEGNGSGGNGSGGNGSGN',\n",
" 'label': 0.7599999904632561}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset[\"train\"][0]"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6e10dbbb-73ef-4b67-8290-77f8896298f5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"datasets mean token lenght 17.24006958538707 min token length 12 max token length 35\n"
]
}
],
"source": [
"token_len_list = []\n",
"for item in dataset[\"test\"]:\n",
" inputs = tokenizer.tokenize(item[\"seq\"])\n",
" token_len_list.append( len(inputs) )\n",
"\n",
"mean_len = sum(token_len_list)/len(token_len_list)\n",
"min_len = min(token_len_list)\n",
"max_len = max(token_len_list)\n",
"\n",
"print(\"datasets \", \"mean token lenght\", mean_len, \"min token length\", min_len, \"max token length\", max_len)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "ac58b5b4-bff0-404d-bcf5-2b93db2b37c0",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "419cce8c5ba249ac8c8773dd2d69992d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/62079 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Asking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.\n",
"Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0b9ea09fe3ea49b19f7d52aca7949acf",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/6898 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# 2. tokenize\n",
"def tokenize_function(examples):\n",
" return tokenizer(examples['seq'], truncation=True, padding='max_length')\n",
"\n",
"# 3. 对数据集应用分词函数\n",
"tokenized_datasets = dataset.map(tokenize_function, batched=True)\n",
"\n",
"# 4. 创建一个数据收集器，用于动态填充和遮蔽\n",
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "94f6d643-2cf7-4651-9a8d-1884b2bddd1c",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/transformers/training_args.py:1575: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
" warnings.warn(\n",
"/tmp/ipykernel_1347/4285456223.py:23: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n",
" trainer = Trainer(\n"
]
}
],
"source": [
"from transformers import TrainingArguments, Trainer\n",
"import numpy as np\n",
"from sklearn.metrics import mean_squared_error\n",
"\n",
"\n",
"def compute_metrics(eval_pred):\n",
" predictions, labels = eval_pred\n",
" rmse = mean_squared_error(labels, predictions)\n",
" return {\"rmse\": rmse}\n",
"\n",
"# 设置训练参数\n",
"training_args = TrainingArguments(\n",
" output_dir='./results',\n",
" evaluation_strategy=\"epoch\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=20,\n",
" per_device_eval_batch_size=20,\n",
" num_train_epochs=10,\n",
" weight_decay=0.01,\n",
")\n",
"\n",
"# 使用Trainer API进行训练（假设已有train_dataset和eval_dataset）\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"test\"],\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dfe12979-d977-4404-bf9e-18c1f91a3e39",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='30987' max='31040' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [30987/31040 1:00:56 < 00:06, 8.47 it/s, Epoch 9.98/10]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" <th>Rmse</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.044600</td>\n",
" <td>0.163462</td>\n",
" <td>0.163462</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.041900</td>\n",
" <td>0.157900</td>\n",
" <td>0.157900</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.037700</td>\n",
" <td>0.159724</td>\n",
" <td>0.159724</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4</td>\n",
" <td>0.031700</td>\n",
" <td>0.157686</td>\n",
" <td>0.157686</td>\n",
" </tr>\n",
" <tr>\n",
" <td>5</td>\n",
" <td>0.028800</td>\n",
" <td>0.157124</td>\n",
" <td>0.157124</td>\n",
" </tr>\n",
" <tr>\n",
" <td>6</td>\n",
" <td>0.025400</td>\n",
" <td>0.150852</td>\n",
" <td>0.150852</td>\n",
" </tr>\n",
" <tr>\n",
" <td>7</td>\n",
" <td>0.022300</td>\n",
" <td>0.159293</td>\n",
" <td>0.159293</td>\n",
" </tr>\n",
" <tr>\n",
" <td>8</td>\n",
" <td>0.019600</td>\n",
" <td>0.154608</td>\n",
" <td>0.154608</td>\n",
" </tr>\n",
" <tr>\n",
" <td>9</td>\n",
" <td>0.017300</td>\n",
" <td>0.156104</td>\n",
" <td>0.156104</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"IOPub message rate exceeded.\n",
"The Jupyter server will temporarily stop sending output\n",
"to the client in order to avoid crashing it.\n",
"To change this limit, set the config variable\n",
"`--ServerApp.iopub_msg_rate_limit`.\n",
"\n",
"Current values:\n",
"ServerApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
"ServerApp.rate_limit_window=3.0 (secs)\n",
"\n"
]
}
],
"source": [
"# 开始训练\n",
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "060c4618-40d0-4934-bab8-36aab3a46de5",
"metadata": {},
"outputs": [],
"source": [
"#模型测试\n",
"predictions = trainer.predict(tokenized_datasets[\"test\"])\n",
"predictions"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "1f8ef885-5bc9-4668-905b-6b2235209654",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='345' max='345' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [345/345 00:09]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.15949687361717224,\n",
" 'eval_rmse': 0.15949687361717224,\n",
" 'eval_runtime': 9.1483,\n",
" 'eval_samples_per_second': 754.017,\n",
" 'eval_steps_per_second': 37.712,\n",
" 'epoch': 10.0}"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.evaluate()"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "afabdbe9-9b96-4f9e-bef2-1d819431f8d1",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[ 1.7208484 ]\n",
" [ 0.00225139]\n",
" [ 0.3325616 ]\n",
" [-0.34372616]\n",
" [-0.45505935]\n",
" [-0.06892765]\n",
" [ 0.15099108]\n",
" [ 0.12211376]\n",
" [ 0.3947332 ]\n",
" [ 0.23186803]]\n"
]
}
],
"source": [
"predictions.predictions[0:10].squeeze()"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "fa9d17fd-eece-4c1e-99e0-3d19d36f7584",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 1.69, 0.84, 0.58, -0.15, 0.23, 0.03, 0.15, 0.2 , 0.51,\n",
" 1.1 ], dtype=float32)"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predictions.label_ids[0:10]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "52252015-e068-414b-bd8a-79a5d1a2beec",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03-gene-task/data/.ipynb_checkpoints/protein_stab-checkpoint.csv
+68978
View File
File diff suppressed because it is too large Load Diff
03-gene-task/data/dna_protein_full.json
+16000
View File
File diff suppressed because it is too large Load Diff
03-gene-task/data/protein_stab.csv
+68978
View File
File diff suppressed because it is too large Load Diff
03-gene-task/env_ini.ipynb
+267
View File
@@ -0,0 +1,267 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 5,
"id": "e3fbdac5-cd38-4e41-b5d2-d9d112b4ac1b",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
"Requirement already satisfied: transformers in /root/miniconda3/lib/python3.12/site-packages (4.47.1)\n",
"Requirement already satisfied: sentencepiece in /root/miniconda3/lib/python3.12/site-packages (0.2.0)\n",
"Requirement already satisfied: google in /root/miniconda3/lib/python3.12/site-packages (3.0.0)\n",
"Requirement already satisfied: protobuf in /root/miniconda3/lib/python3.12/site-packages (5.27.0)\n",
"Requirement already satisfied: deepspeed in /root/miniconda3/lib/python3.12/site-packages (0.16.2)\n",
"Requirement already satisfied: peft in /root/miniconda3/lib/python3.12/site-packages (0.14.0)\n",
"Collecting datasets\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d7/84/0df6c5981f5fc722381662ff8cfbdf8aad64bec875f75d80b55bfef394ce/datasets-3.2.0-py3-none-any.whl (480 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m480.6/480.6 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: filelock in /root/miniconda3/lib/python3.12/site-packages (from transformers) (3.14.0)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.27.0)\n",
"Requirement already satisfied: numpy>=1.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (1.26.4)\n",
"Requirement already satisfied: packaging>=20.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (23.2)\n",
"Requirement already satisfied: pyyaml>=5.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (6.0.1)\n",
"Requirement already satisfied: regex!=2019.12.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2024.11.6)\n",
"Requirement already satisfied: requests in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2.31.0)\n",
"Requirement already satisfied: tokenizers<0.22,>=0.21 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.21.0)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.4.5)\n",
"Requirement already satisfied: tqdm>=4.27 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (4.66.2)\n",
"Requirement already satisfied: beautifulsoup4 in /root/miniconda3/lib/python3.12/site-packages (from google) (4.12.3)\n",
"Requirement already satisfied: einops in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (0.8.0)\n",
"Requirement already satisfied: hjson in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (3.1.0)\n",
"Requirement already satisfied: msgpack in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.1.0)\n",
"Requirement already satisfied: ninja in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.11.1.3)\n",
"Requirement already satisfied: psutil in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (5.9.8)\n",
"Requirement already satisfied: py-cpuinfo in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (9.0.0)\n",
"Requirement already satisfied: pydantic>=2.0.0 in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.10.4)\n",
"Requirement already satisfied: torch in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.3.0+cu121)\n",
"Requirement already satisfied: nvidia-ml-py in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (12.560.30)\n",
"Requirement already satisfied: accelerate>=0.21.0 in /root/miniconda3/lib/python3.12/site-packages (from peft) (1.2.1)\n",
"Collecting pyarrow>=15.0.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/3a/2e/3b99f8a3d9e0ccae0e961978a0d0089b25fb46ebbcfb5ebae3cca179a5b3/pyarrow-18.1.0-cp312-cp312-manylinux_2_28_x86_64.whl (40.1 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.1/40.1 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting dill<0.3.9,>=0.3.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/c9/7a/cef76fd8438a42f96db64ddaa85280485a9c395e7df3db8158cfec1eee34/dill-0.3.8-py3-none-any.whl (116 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m53.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting pandas (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/38/f8/d8fddee9ed0d0c0f4a2132c1dfcf0e3e53265055da8df952a53e7eaf178c/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.7/12.7 MB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting requests (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tqdm>=4.27 (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting xxhash (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/a7/81dba5010f7e733de88af9555725146fc133be97ce36533867f4c7e75066/xxhash-3.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.4/194.4 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multiprocess<0.70.17 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/0a/7d/a988f258104dcd2ccf1ed40fdc97e26c4ac351eeaf81d76e266c52d84e2f/multiprocess-0.70.16-py312-none-any.whl (146 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.7/146.7 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /root/miniconda3/lib/python3.12/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.5.0)\n",
"Collecting aiohttp (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/40/7f/6de218084f9b653026bd7063cd8045123a7ba90c25176465f266976d8c82/aiohttp-3.11.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting aiohappyeyeballs>=2.3.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/b9/74/fbb6559de3607b3300b9be3cc64e97548d55678e44623db17820dbd20002/aiohappyeyeballs-2.4.4-py3-none-any.whl (14 kB)\n",
"Collecting aiosignal>=1.1.2 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/ec/6a/bc7e17a3e87a2985d3e8f4da4cd0f481060eb78fb08596c42be62c90a4d9/aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)\n",
"Requirement already satisfied: attrs>=17.3.0 in /root/miniconda3/lib/python3.12/site-packages (from aiohttp->datasets) (23.2.0)\n",
"Collecting frozenlist>=1.1.1 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/af/f2/64b73a9bb86f5a89fb55450e97cd5c1f84a862d4ff90d9fd1a73ab0f64a5/frozenlist-1.5.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m283.6/283.6 kB\u001b[0m \u001b[31m41.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multidict<7.0,>=4.5 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d3/c8/529101d7176fe7dfe1d99604e48d69c5dfdcadb4f06561f465c8ef12b4df/multidict-6.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (131 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m131.0/131.0 kB\u001b[0m \u001b[31m56.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting propcache>=0.2.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1c/07/ebe102777a830bca91bbb93e3479cd34c2ca5d0361b83be9dbd93104865e/propcache-0.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (243 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.6/243.6 kB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting yarl<2.0,>=1.17.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1a/e1/a097d5755d3ea8479a42856f51d97eeff7a3a7160593332d98f2709b3580/yarl-1.18.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (336 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m336.9/336.9 kB\u001b[0m \u001b[31m41.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /root/miniconda3/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.24.0->transformers) (4.12.2)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.2 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (2.27.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.0.4)\n",
"Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (3.7)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.1.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2024.2.2)\n",
"Requirement already satisfied: sympy in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (1.12.1)\n",
"Requirement already satisfied: networkx in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.3)\n",
"Requirement already satisfied: jinja2 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.1.4)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (8.9.2.26)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.3.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.0.2.54)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (10.3.2.106)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.4.5.107)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.0.106)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (2.20.5)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/miniconda3/lib/python3.12/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->deepspeed) (12.5.40)\n",
"Requirement already satisfied: soupsieve>1.2 in /root/miniconda3/lib/python3.12/site-packages (from beautifulsoup4->google) (2.5)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /root/miniconda3/lib/python3.12/site-packages (from pandas->datasets) (2.9.0.post0)\n",
"Collecting pytz>=2020.1 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/c3/005fcca25ce078d2cc29fd559379817424e94885510568bc1bc53d7d5846/pytz-2024.2-py2.py3-none-any.whl (508 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m508.0/508.0 kB\u001b[0m \u001b[31m38.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tzdata>=2022.7 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/a6/ab/7e5f53c3b9d14972843a647d8d7a853969a58aecc7559cb3267302c94774/tzdata-2024.2-py2.py3-none-any.whl (346 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m346.6/346.6 kB\u001b[0m \u001b[31m36.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: six>=1.5 in /root/miniconda3/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/lib/python3.12/site-packages (from jinja2->torch->deepspeed) (2.1.5)\n",
"Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /root/miniconda3/lib/python3.12/site-packages (from sympy->torch->deepspeed) (1.3.0)\n",
"Installing collected packages: pytz, xxhash, tzdata, tqdm, requests, pyarrow, propcache, multidict, frozenlist, dill, aiohappyeyeballs, yarl, pandas, multiprocess, aiosignal, aiohttp, datasets\n",
" Attempting uninstall: tqdm\n",
" Found existing installation: tqdm 4.66.2\n",
" Uninstalling tqdm-4.66.2:\n",
" Successfully uninstalled tqdm-4.66.2\n",
" Attempting uninstall: requests\n",
" Found existing installation: requests 2.31.0\n",
" Uninstalling requests-2.31.0:\n",
" Successfully uninstalled requests-2.31.0\n",
"Successfully installed aiohappyeyeballs-2.4.4 aiohttp-3.11.11 aiosignal-1.3.2 datasets-3.2.0 dill-0.3.8 frozenlist-1.5.0 multidict-6.1.0 multiprocess-0.70.16 pandas-2.2.3 propcache-0.2.1 pyarrow-18.1.0 pytz-2024.2 requests-2.32.3 tqdm-4.67.1 tzdata-2024.2 xxhash-3.5.0 yarl-1.18.3\n",
"\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"!pip install transformers sentencepiece google protobuf deepspeed peft datasets "
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "4e906370-40c7-4f6b-a700-f183a9308c78",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"https://hf-mirror.com\n"
]
}
],
"source": [
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ecc98529-6581-41d2-a876-23ce5187cae7",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b01fc372-33af-46e5-8c0e-8bccba7237ee",
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"# load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"dnagpt/dna_core_promoter\")['train'].train_test_split(test_size=0.1)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "136c38d4-bd0f-4ecd-9165-2fd5b5207c1d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 53276\n",
" })\n",
" test: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 5920\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "markdown",
"id": "28acb64e-8d1e-4482-a515-344a2e0344c4",
"metadata": {},
"source": [
"## lfs 支持\n",
"apt-get update\n",
"\n",
"apt-get install git-lfs\n",
"\n",
"git lfs install"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d3cefb0-1eed-4f23-8591-1990f7113820",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
03-gene-task/img/.ipynb_checkpoints/dataset-checkpoint.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 253 KiB

03-gene-task/img/2_structure.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 115 KiB

03-gene-task/img/dataset.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 253 KiB

03-gene-task/img/ds_structure.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 272 KiB

03-gene-task/img/function.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 343 KiB

03-gene-task/img/gpt2-ft.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 224 KiB

03-gene-task/img/pdb1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 135 KiB

03-gene-task/img/protein-structure-1-2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.0 MiB

03-gene-task/img/protein-structure-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 829 KiB

03-gene-task/img/protein-structure-2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 555 KiB

03-gene-task/img/sequence.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 186 KiB

04-gene-sft/.ipynb_checkpoints/1-finetue-intro-checkpoint.ipynb
+384
View File
@@ -0,0 +1,384 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "32216f81-0979-4afd-8c8c-16729cd0dab6",
"metadata": {},
"source": [
"# 4.1 模型微调VS指令微调"
]
},
{
"cell_type": "markdown",
"id": "7cd7f9b2-c0a3-48c2-848e-a1e9c7188f03",
"metadata": {},
"source": [
"## 一个典型的知乎问题\n",
"\n",
"### **问题**\n",
"\n",
"用LLM实现文本二分类，微调base模型还是微调chat模型比较好？[问题](https://www.zhihu.com/question/632473480/answer/38930949853)\n",
"\n",
"我想用开源LLM（例如chatglm，baichuan）实现文本二分类（比如正负情感分类），有一组训练数据可以用于微调模型，提升分类性能，这时候应该选择base模型还是chat模型？\n",
"\n",
"\n",
"### **回答**\n",
"1 如果是使用2分类的header，base模型好一些。\n",
"\n",
"也就是使用如下类似的的设置。\n",
"\n",
"model = AutoModelForSequenceClassification.from_pretrained(\n",
"\"yuanzhoulvpi/gpt2_chinese\", num_labels=2\n",
")\n",
"\n",
"对应的训练数据一般是这样的：\n",
"\n",
"| seq | label |\n",
"|------------------------------|-------|\n",
"| 他家的奶茶超级好喝。。。 | 1 |\n",
"| 他家的奶茶超级难喝。。。 | 0 |\n",
"\n",
"\n",
"2 如果是把分类问题，改成指令微调的模式，就是像\n",
"\n",
"```\n",
"{\n",
"\n",
"\"instruction\": \"你现在在做一项情感分类的任务，如果是积极情感，则回答积极。消极情感则回答消极。\"\n",
"\"input\"：他家的奶茶超级好喝。。。\n",
"\"output\"：“积极”\n",
"\n",
"}\n",
"```\n",
"\n",
"然后进行指令微调，lora/peft调整部分参数就行，一般是chat模型比较好。\n",
"\n",
"\n",
"\n",
"这种二分类问题，用llm就是大材小用了，一般就是选个小的的模型，用AutoModelForSequenceClassification效果最好，如果追求SOTA，有些研究表明搞成指令微调模式效果可能更好。"
]
},
{
"cell_type": "markdown",
"id": "2cfcc1e9-ddda-4a1c-871b-0508fd421ed5",
"metadata": {},
"source": [
"## 大模型微调（Fine-tuning）和指令微调（Instruction Tuning）\n",
"\n",
"普通的大模型微调（Fine-tuning）和指令微调（Instruction Tuning）是两种不同的训练方法，它们适用于不同的应用场景，并且在实现细节上也有所区别。\n",
"\n",
"\n",
"#### 1. **定义**\n",
"\n",
"普通微调是指在一个预训练好的大模型基础上，针对特定任务添加一个或多个新层（通常称为头部或 header），然后使用特定任务的数据集对整个模型（包括新添加的层）进行再训练的过程。对于分类任务，常见的做法是在 GPT-2 的顶部添加一个分类头。\n",
"\n",
"#### 2. **具体步骤**\n",
"\n",
"- **添加分类头**：为 GPT-2 添加一个分类头，该头通常包含线性层（全连接层）以及可能的激活函数和归一化层。\n",
" \n",
"- **准备数据**：准备好用于微调的任务特定数据集，如文本分类、情感分析等。\n",
" \n",
"- **微调过程**：\n",
" - 使用任务特定的数据集对整个模型（包括预训练权重和新添加的分类头）进行再训练。\n",
" - 通常会调整学习率、批量大小等超参数以优化性能。\n",
" - 可能只对新添加的层进行训练，或者对整个模型进行微调（取决于资源和需求）。\n",
"\n",
"#### 3. **适用场景**\n",
"\n",
"- **任务明确**：当有清晰的任务目标时，例如文本分类、命名实体识别等。\n",
"- **标签数据可用**：拥有足够的标注数据来进行监督学习。\n",
"\n",
"#### 4. **优点**\n",
"\n",
"- **针对性强**：能够有效地提升模型在特定任务上的表现。\n",
"- **资源利用效率高**：相比于从头开始训练，微调需要的计算资源和时间较少。\n",
"\n",
"#### 5. **缺点**\n",
"\n",
"- **泛化能力有限**：微调后的模型可能在未见过的任务或领域中表现不佳。\n",
"\n",
"### 指令微调（Instruction Tuning）\n",
"\n",
"#### 1. **定义**\n",
"\n",
"指令微调是一种更为通用的微调方法，它旨在让模型理解和遵循自然语言指令，而不是直接针对某个特定任务进行优化。这种方法通过提供一系列指令-输出对来训练模型，使其学会根据指令生成适当的响应。\n",
"\n",
"#### 2. **具体步骤**\n",
"\n",
"- **构造指令数据集**：创建一个包含各种指令及其预期输出的数据集。这些指令可以覆盖多种任务类型，如问答、翻译、摘要生成等。\n",
" \n",
"- **微调过程**：\n",
" - 使用指令数据集对模型进行训练，使模型能够理解并执行不同类型的指令。\n",
" - 强调模型对自然语言指令的理解和执行，而非特定于某一任务的优化。\n",
"\n",
"#### 3. **适用场景**\n",
"\n",
"- **多任务适应**：当希望模型能够在多种不同类型的任务中表现出色时。\n",
"- **少样本学习**：在仅有少量示例的情况下，仍然可以让模型快速适应新任务。\n",
"\n",
"#### 4. **优点**\n",
"\n",
"- **灵活性高**：模型可以在没有额外训练的情况下处理新的任务。\n",
"- **跨领域泛化能力强**：更有可能在未曾见过的任务或领域中保持良好的性能。\n",
"\n",
"#### 5. **缺点**\n",
"\n",
"- **复杂度增加**：指令微调通常涉及更多的训练数据和更复杂的训练过程。\n",
"- **评估难度较大**：由于任务的多样性，评估模型性能变得更加困难。\n",
"\n",
"\n",
"### 小结\n",
"\n",
"普通微调侧重于提高模型在特定任务上的性能，而指令微调则更加注重模型对自然语言指令的理解和执行能力。选择哪种方法取决于你的具体需求和应用场景。如果你有一个明确的任务并且有大量的标注数据，那么普通微调可能是更好的选择；如果你希望模型具有更高的灵活性和跨任务适应能力，则可以考虑指令微调。"
]
},
{
"cell_type": "markdown",
"id": "6203be53-18a5-447d-9071-32e031934b9c",
"metadata": {},
"source": [
"## 从GPT到chatGPT\n",
"\n",
"关键点在于指令微调（Instruction Tuning）\n",
"* 将所有任务统一为指令形式\n",
"* 多任务精调\n",
"* 与人类对齐（多样性）\n",
"* 进一步分为有监督指令微调和带有人类反馈的强化学习(RLHF)\n",
"\n",
"告别微调\n",
"\n",
"因为GPT-3使用了天量级的数据来进行预训练，所以学到的知识也更多更通用，以致于GPT-3打出的口号就是“告别微调的GPT-3”。\n",
"\n",
"相比于BERT这种预训练+微调的两阶段模型，GPT-3的目标是模型更加通用，从而解决BERT这种下游任务微调需要依赖领域标注数据的情况。\n",
"\n",
"拿我们实际业务举例，我主要做分本分类任务。对于使用BERT来完成文本分类任务来说，首先我需要使用海量的无标注文本数据进行预训练学习语言学知识。\n",
"\n",
"幸运的是这种预训练过程一般是一次性的，训练完成后可以把模型保存下来继续使用。很多大厂比如谷歌、Facebook等把得到的预训练模型开源了出来，所以咱们只需要导入预训练好的模型权重就可以直接使用了，相当于完成了模型的预训练过程；第二阶段就是微调了，对于文本分类等下游任务来说， 我们需要一批带标签的训练语料来微调模型。不同的下游任务会需要特定的训练语料。这时候面临的一个最大的问题是训练语料是需要人工标注的，而标注的成本是非常高的。除此之外不同的标注人员因为经验阅历等不同导致对同一条文本的理解也不同，所以容易出现标注不一致的问题。当标注数据量较少时还容易出现模型过拟合。归根结底就是微调是需要标注数据的，而获取标注数据的成本是很高的。\n",
"\n",
"为了解决这个问题，GPT-3可以让NLPer不用标注训练语料就能很好的完成下游任务，让GPT-3更通用更便利。GPT-3不需要进行微调的结构图如下所示：\n",
"\n",
"<img src='img/sft.png' width='600px' />"
]
},
{
"cell_type": "markdown",
"id": "28e037df-734b-4fe7-ac07-311f1b3a7d7b",
"metadata": {},
"source": [
"## 指令微调数据构建\n",
"\n",
"<img src='img/sft2.png' width='800px' />\n",
"\n",
"\n",
"\n",
"根据典型的分类语料数据，构建指令微调数据\n",
"\n",
"目前如llama等都使用Alpaca格式\n",
"\n",
"指令数据当做一般的文本，进行无监督的训练，和预训练流程一致"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "64312191-423f-4a18-aa0c-036374e93fb2",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "32c16282-f9f1-4545-b522-daf2b39b4ead",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
")"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#原始模型\n",
"from transformers import AutoModel\n",
"model = AutoModel.from_pretrained(\"gpt2\")\n",
"model"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1149163f-4d89-472e-8d45-ebcbb5f9575e",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at gpt2 and are newly initialized: ['score.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
},
{
"data": {
"text/plain": [
"GPT2ForSequenceClassification(\n",
" (transformer): GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" )\n",
" (score): Linear(in_features=768, out_features=2, bias=False)\n",
")"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#分类微调模型\n",
"from transformers import AutoModelForSequenceClassification\n",
"ft_model = AutoModelForSequenceClassification.from_pretrained(\"gpt2\", num_labels=2)\n",
"ft_model"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "09735059-507c-48c4-893f-ca0da21ce5e8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GPT2LMHeadModel(\n",
" (transformer): GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" )\n",
" (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
")"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#指令微调模型\n",
"from transformers import AutoModelForCausalLM\n",
"sft_model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n",
"sft_model"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d1407cbe-4996-4898-a135-e26d28da2a2a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/2-gpt2-instruction-ft-checkpoint.ipynb
+1688
View File
File diff suppressed because it is too large Load Diff
04-gene-sft/.ipynb_checkpoints/3-llama-expand-dict-checkpoint.ipynb
+339
View File
@@ -0,0 +1,339 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "6bad311a-c949-4246-9e6b-6d4ec76699b7",
"metadata": {},
"source": [
"# 4.3 基于llama的基因数据词典扩充"
]
},
{
"cell_type": "markdown",
"id": "d42860cf-14fc-48f5-ac6c-1fd92a6a92ba",
"metadata": {},
"source": [
"前面介绍了huggingface自带的分词器构建代码，这里介绍下更为通用的sentencepiece，部分huggingface其实就是来自于这个框架。\n",
"\n",
"SentencePiece 是一个语言无关的分词框架，由 Google 开发并开源。它不同于传统的基于词汇表（如词典）的分词方法，而是采用一种无监督的学习方式来训练模型，从而将文本分割成“子词”单元（subword units）。这种方法使得 SentencePiece 在处理未知词、罕见词以及多语言文本时表现出色。\n",
"\n",
"### 主要特点\n",
"\n",
"1. **语言无关**：\n",
" - SentencePiece 不依赖于任何特定语言的规则或词典，因此它可以应用于任何语言，甚至是混合语言的文本。\n",
"\n",
"2. **子词分词**：\n",
" - 它生成的是子词级别的 token，而不是完整的单词。这种方式可以有效地处理 OOV (out-of-vocabulary) 问题，并且有助于减少词汇表的大小。\n",
"\n",
"3. **无监督学习**：\n",
" - SentencePiece 使用无监督的方法从原始文本中学习分词规则，这意味着你只需要提供未标注的文本数据即可训练分词模型。\n",
"\n",
"4. **灵活的分词粒度**：\n",
" - 可以通过调整参数控制分词的粒度，即生成的子词单元的平均长度。这允许根据具体应用需求优化性能。\n",
"\n",
"5. **支持 BPE 和 Unigram LM**：\n",
" - SentencePiece 实现了两种流行的分词算法：Byte Pair Encoding (BPE) 和 Unigram Language Model (Unigram LM)。这两种方法各有优劣，可以根据任务选择合适的一种。\n",
"\n",
"6. **易于集成**：\n",
" - 提供了多种编程语言的绑定，包括 Python、C++、Go 等，方便在不同环境中使用。\n",
"\n",
"### 工作流程\n",
"\n",
"1. **准备语料库**：\n",
" - 收集用于训练分词模型的未标注文本数据。\n",
"\n",
"2. **训练模型**：\n",
" - 使用 `sentencepiece_trainer` 工具对收集到的文本进行训练，生成分词模型文件。\n",
" ```bash\n",
" spm_train --input=your_corpus.txt --model_prefix=myprefix --vocab_size=8000\n",
" ```\n",
"\n",
"3. **编码和解码**：\n",
" - 训练完成后，可以使用生成的模型对新文本进行编码（分词）和解码（还原）。\n",
" ```python\n",
" import sentencepiece as spm\n",
"\n",
" # 加载训练好的模型\n",
" sp = spm.SentencePieceProcessor(model_file='myprefix.model')\n",
"\n",
" # 分词\n",
" encoded = sp.encode(\"这是一个测试句子。\", out_type=str)\n",
" print(encoded)\n",
"\n",
" # 还原\n",
" decoded = sp.decode(encoded)\n",
" print(decoded)\n",
" ```\n",
"\n",
"### 应用场景\n",
"\n",
"- **自然语言处理 (NLP)**：广泛应用于各种 NLP 任务，如机器翻译、文本分类、情感分析等。\n",
"- **多语言支持**：特别适合处理包含多种语言的文本。\n",
"- **低资源语言**：对于那些缺乏丰富词汇资源的语言尤其有用。\n",
"- **预训练语言模型**：许多现代预训练语言模型（如 BERT、T5、mBART）都采用了 SentencePiece 作为其分词工具。\n",
"\n",
"### 小结\n",
"\n",
"SentencePiece 是一个强大而灵活的分词框架，适用于广泛的文本处理任务。它的无监督学习特性、语言无关性和高效的子词分词能力使其成为处理复杂和多样化文本数据的理想选择。希望这个简单的介绍能帮助你理解 SentencePiece 的基本概念和应用场景。如果有更多问题或需要进一步的帮助，请随时提问！"
]
},
{
"cell_type": "markdown",
"id": "a8dedb50-a428-4146-8edf-84e699abf81b",
"metadata": {},
"source": [
"## GENE分词器构建"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39b5bf12-eaf0-432e-a2b0-99ba437daf3e",
"metadata": {},
"outputs": [],
"source": [
"!pip install sentencepiece"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3b732b8e-53d1-4bfa-891b-2d63b886cc4a",
"metadata": {},
"outputs": [],
"source": [
"import sentencepiece as spm\n",
"\n",
"spm.SentencePieceTrainer.train(input='../01-data_env/data/dna_1g.txt,../01-data_env/data/protein_1g.txt',\n",
" model_prefix='gene_bpe_seg', \n",
" vocab_size=60000,\n",
" model_type='bpe', #默认是unigram\n",
" num_threads=10,\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "19a06b82-31b8-48cb-9c83-ec016da2da8a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['▁TCG', 'ACGGC', 'ACGCG', 'ACAGC', 'AGCG', 'AGCCCC', 'GCGC', 'ACCCG', 'AGCGCG', 'AKCG', 'FVGP', 'MV', 'HLKV', 'HLE', 'ADV', 'ASSC', 'RS', 'AVI', 'YL', 'TS', 'EEP', 'FEG', 'VLGL', 'RLKE', 'GI', 'AI', 'TGC', 'WPR', 'WP', 'DEM', 'DE', 'RS', 'AVW', 'RV', 'EPY', 'TR', 'HFG', 'RVL', 'YS', 'FGV']\n"
]
}
],
"source": [
"from sentencepiece import SentencePieceProcessor\n",
"model_path = \"gene_bpe_seg.model\"\n",
"sp_model = SentencePieceProcessor(model_file=model_path)\n",
"mm = sp_model.EncodeAsPieces(\"TCGACGGCACGCGACAGCAGCGAGCCCCGCGCACCCGAGCGCGAKCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKEGIAITGCWPRWPDEMDERSAVWRVEPYTRHFGRVLYSFGV\")\n",
"print(mm)"
]
},
{
"cell_type": "markdown",
"id": "958f7bd6-060f-48f4-8afe-02c3032312eb",
"metadata": {},
"source": [
"## 合并词典到llama\n",
"\n",
"我们以基础版本的llama为例，进行合并，请注意llama的使用限制。\n",
"\n",
"新版本的llama需要自行认证下载。[链接](https://huggingface.co/meta-llama)\n",
"\n",
"```\n",
"#建议在终端下执行\n",
"pip install -U huggingface_hub\n",
"export HF_ENDPOINT=https://hf-mirror.com\n",
"huggingface-cli download --resume-download yahma/llama-7b-hf --local-dir llama-7b-hf\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "3bafcc33-2923-4026-bc39-c6ec716d2e3c",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION\"]=\"python\"\n",
"from transformers import LlamaTokenizer\n",
"from sentencepiece import sentencepiece_model_pb2 as sp_pb2_model\n",
"import sentencepiece as spm"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "66cb86ed-3225-4bb0-8aca-6005bc918d03",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"32000 60000\n",
"['<s>', '</s>', '<unk>']\n",
"[1, 2, 0]\n",
"{'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>'}\n"
]
}
],
"source": [
"llama_tokenizer_dir = \"llama-7b-hf\" \n",
"dna_sp_model_file = \"gene_bpe_seg.model\"\n",
"\n",
"# load\n",
"llama_tokenizer = LlamaTokenizer.from_pretrained(llama_tokenizer_dir)\n",
"dna_sp_model = spm.SentencePieceProcessor()\n",
"dna_sp_model.Load(dna_sp_model_file)\n",
"\n",
"llama_spm = sp_pb2_model.ModelProto()\n",
"llama_spm.ParseFromString(llama_tokenizer.sp_model.serialized_model_proto())\n",
"dna_spm = sp_pb2_model.ModelProto()\n",
"dna_spm.ParseFromString(dna_sp_model.serialized_model_proto())\n",
"\n",
"# print number of tokens\n",
"print(len(llama_tokenizer),len(dna_sp_model))\n",
"print(llama_tokenizer.all_special_tokens)\n",
"print(llama_tokenizer.all_special_ids)\n",
"print(llama_tokenizer.special_tokens_map)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7ba4240e-bc08-4be0-8ca3-c4e7a47fa055",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"32000\n",
"Before:32000\n",
"New model pieces: 91643\n"
]
}
],
"source": [
"## Add dna tokens to LLaMA tokenizer\n",
"llama_spm_tokens_set=set(p.piece for p in llama_spm.pieces)\n",
"print(len(llama_spm_tokens_set))\n",
"print(f\"Before:{len(llama_spm_tokens_set)}\")\n",
"for p in dna_spm.pieces:\n",
" piece = p.piece\n",
" score = p.score\n",
" if piece not in llama_spm_tokens_set:\n",
" new_p = sp_pb2_model.ModelProto().SentencePiece()\n",
" new_p.piece = piece\n",
" new_p.score = score # 0?\n",
" llama_spm.pieces.append(new_p)\n",
"print(f\"New model pieces: {len(llama_spm.pieces)}\")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a240a7d8-c1a9-4473-a5c5-157a25f97c16",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"gene-LLaMA tokenizer has been saved to merged_gene_eng_tokenizer_hf\n"
]
}
],
"source": [
"## Save\n",
"output_sp_dir = 'merged_gene_eng_tokenizer_sp'\n",
"output_hf_dir = 'merged_gene_eng_tokenizer_hf' # the path to save dna-LLaMA tokenizer\n",
"os.makedirs(output_sp_dir,exist_ok=True)\n",
"with open(output_sp_dir+'/gene_eng_llama_tokenizer.model', 'wb') as f:\n",
" f.write(llama_spm.SerializeToString())\n",
"\n",
"tokenizer = LlamaTokenizer(vocab_file=output_sp_dir+'/gene_eng_llama_tokenizer.model')\n",
"tokenizer.save_pretrained(output_hf_dir)\n",
"print(f\"gene-LLaMA tokenizer has been saved to {output_hf_dir}\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "cbd1f648-f8a0-4f16-b516-2ce3e7c7cfee",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['<s>', '</s>', '<unk>']\n",
"[1, 2, 0]\n",
"{'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>'}\n",
"Test text:\n",
" TCGACGGCACGCGACAGCAGCGAGCCCCGCGCACCCGAGCGCGAKCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKEGIAITGCWPRWPDEMDERSAVWRVEPYTRHFGRVLYSFGV,\n",
"The primary use of LLaMA is research on large language models, including\n",
"Tokenized by LLaMA tokenizer:['▁T', 'CG', 'AC', 'G', 'GC', 'AC', 'GC', 'G', 'AC', 'AG', 'CA', 'GC', 'G', 'AG', 'CC', 'CC', 'GC', 'GC', 'AC', 'CC', 'GA', 'GC', 'GC', 'GA', 'K', 'CG', 'F', 'V', 'G', 'PM', 'V', 'HL', 'K', 'V', 'H', 'LE', 'AD', 'VA', 'SS', 'CR', 'S', 'AV', 'I', 'Y', 'LT', 'SEE', 'PF', 'EG', 'V', 'L', 'GL', 'RL', 'KE', 'G', 'IA', 'IT', 'GC', 'W', 'PR', 'WP', 'DE', 'MD', 'ERS', 'AV', 'WR', 'VE', 'PY', 'TR', 'H', 'F', 'GR', 'V', 'LY', 'SF', 'GV', ',', '<0x0A>', 'The', '▁primary', '▁use', '▁of', '▁L', 'La', 'MA', '▁is', '▁research', '▁on', '▁large', '▁language', '▁models', ',', '▁including']\n",
"Tokenized by GENE-LLaMA tokenizer:['▁TCG', 'ACGGC', 'ACGCG', 'ACAG', 'CA', 'GCG', 'AGCCCC', 'GCGC', 'ACCCG', 'AGCGCG', 'AKCG', 'FVGP', 'MVHL', 'KV', 'HLE', 'ADV', 'ASSC', 'RSAV', 'I', 'YL', 'TSEE', 'P', 'FEG', 'VLGL', 'RLK', 'EGI', 'AI', 'TGC', 'W', 'PRW', 'P', 'DEM', 'DER', 'SAV', 'W', 'RVE', 'PY', 'TRH', 'FG', 'RVLY', 'SFGV', ',', '<0x0A>', 'The', '▁primary', '▁use', '▁of', '▁L', 'La', 'MA', '▁is', '▁research', '▁on', '▁large', '▁language', '▁models', ',', '▁including']\n"
]
}
],
"source": [
"# Test\n",
"llama_tokenizer = LlamaTokenizer.from_pretrained(llama_tokenizer_dir)\n",
"dna_llama_tokenizer = LlamaTokenizer.from_pretrained(output_hf_dir)\n",
"print(tokenizer.all_special_tokens)\n",
"print(tokenizer.all_special_ids)\n",
"print(tokenizer.special_tokens_map)\n",
"text='''TCGACGGCACGCGACAGCAGCGAGCCCCGCGCACCCGAGCGCGAKCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKEGIAITGCWPRWPDEMDERSAVWRVEPYTRHFGRVLYSFGV,\n",
"The primary use of LLaMA is research on large language models, including'''\n",
"print(\"Test text:\\n\",text)\n",
"print(f\"Tokenized by LLaMA tokenizer:{llama_tokenizer.tokenize(text)}\")\n",
"print(f\"Tokenized by GENE-LLaMA tokenizer:{dna_llama_tokenizer.tokenize(text)}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "46ae7605-2ef8-4927-bff3-2c0325f8df0d",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/4-deepspeed-intro-checkpoint.ipynb
+619
View File
@@ -0,0 +1,619 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "c2e5c9f4-4378-4d39-bc4f-fb4b4a2b2481",
"metadata": {},
"source": [
"# 4.4 deepspeed分布式训练简介"
]
},
{
"cell_type": "markdown",
"id": "75b8219d-8069-4b18-96c8-d5024ee049f1",
"metadata": {},
"source": [
"## 大模型并行训练简介\n",
"\n",
"大模型的并行训练旨在克服单个 GPU 显存的限制和加速训练过程，通常适用于参数规模较大的模型（如 GPT-3、T5 等）。并行训练主要包括以下几种方法，每种方法适用于不同的场景和模型特性。\n",
"\n",
"---\n",
"\n",
"### **1. 数据并行（Data Parallelism）**\n",
"\n",
"#### **原理**\n",
"- 将数据切分成多个小批次，每个 GPU 处理其中一部分。\n",
"- 模型副本被复制到每个 GPU。\n",
"- 每个 GPU 独立计算梯度，最终通过梯度同步（如 AllReduce 操作）更新参数。\n",
"\n",
"#### **特点**\n",
"- **优点**：\n",
" - 实现简单，是最常用的并行方法。\n",
" - 对模型大小没有限制。\n",
"- **缺点**：\n",
" - 模型副本需要完整加载到每个 GPU，占用显存。\n",
" - 在超大规模模型中，显存压力较大。\n",
"\n",
"#### **适用场景**\n",
"- 参数规模适中，显存可以容纳整个模型的场景。\n",
"\n",
"---\n",
"\n",
"### **2. 模型并行（Model Parallelism）**\n",
"\n",
"#### **原理**\n",
"- 将模型切分成不同的部分，将不同部分分配到不同的 GPU。\n",
"- 前向传播和后向传播时，数据在模型的不同部分之间传递。\n",
"\n",
"#### **特点**\n",
"- **优点**：\n",
" - 不需要复制整个模型，可以支持超大规模模型。\n",
"- **缺点**：\n",
" - GPU 之间通信频繁，可能成为性能瓶颈。\n",
" - 实现复杂，切分模型需要精心设计。\n",
" \n",
"#### **适用场景**\n",
"- 单个 GPU 无法容纳完整模型参数的场景。\n",
"\n",
"#### **具体实现**\n",
"- 将 Transformer 的不同层分配到不同的 GPU。\n",
"- 常用工具：DeepSpeed 的 Pipeline Parallelism、NVIDIA Megatron-LM。\n",
"\n",
"---\n",
"\n",
"### **3. 张量并行（Tensor Parallelism）**\n",
"\n",
"#### **原理**\n",
"- 将模型内部的张量（如权重矩阵）切分为多个子张量，并分配到不同 GPU。\n",
"- GPU 之间协作完成矩阵计算。\n",
"\n",
"#### **特点**\n",
"- **优点**：\n",
" - 减少了每个 GPU 的显存占用，同时保持模型整体完整性。\n",
"- **缺点**：\n",
" - 实现较复杂，需要优化通信操作。\n",
" - 通信开销较高，适合较大批量的训练。\n",
"\n",
"#### **适用场景**\n",
"- 参数非常大的模型（如 GPT-3）。\n",
"- 需要极致优化显存的场景。\n",
"\n",
"#### **具体实现**\n",
"- NVIDIA 的 Megatron-LM 和 Hugging Face Transformers 提供了张量并行的支持。\n",
"\n",
"---\n",
"\n",
"### **4. 管道并行（Pipeline Parallelism）**\n",
"\n",
"#### **原理**\n",
"- 将模型分为不同的部分（通常是按层划分），每部分分配到不同的 GPU。\n",
"- 数据按照流水线的方式流经每个 GPU。\n",
"\n",
"#### **特点**\n",
"- **优点**：\n",
" - 减少每个 GPU 的显存压力。\n",
" - 通过流水线增加计算效率。\n",
"- **缺点**：\n",
" - 引入流水线延迟。\n",
" - 实现复杂，需管理数据依赖和同步。\n",
"\n",
"#### **适用场景**\n",
"- 模型非常深，层数较多的场景。\n",
"\n",
"#### **具体实现**\n",
"- DeepSpeed 的 Pipeline Parallelism。\n",
"\n",
"---\n",
"\n",
"### **5. 混合并行（Hybrid Parallelism）**\n",
"\n",
"#### **原理**\n",
"- 将数据并行、模型并行、张量并行和管道并行组合使用，充分利用多 GPU 资源。\n",
"- 不同的并行方法在不同维度协同工作。\n",
"\n",
"#### **特点**\n",
"- **优点**：\n",
" - 灵活且适应性强，适合超大规模模型。\n",
"- **缺点**：\n",
" - 配置复杂，依赖于框架和训练任务。\n",
"\n",
"#### **适用场景**\n",
"- 超大规模模型（如 GPT-3 或参数量 >1T）。\n",
"- 多机多卡的大型训练环境。\n",
"\n",
"#### **具体实现**\n",
"- NVIDIA Megatron-LM 和 DeepSpeed 的混合并行支持。\n",
"\n",
"---\n",
"\n",
"### **6. ZeRO 优化并行（Zero Redundancy Optimizer）**\n",
"\n",
"#### **原理**\n",
"- 通过分片存储模型参数、优化器状态和梯度，显著减少每个 GPU 的显存占用。\n",
"\n",
"#### **特点**\n",
"- **优点**：\n",
" - 极大降低显存需求。\n",
" - 支持超大规模模型。\n",
"- **缺点**：\n",
" - 对 GPU 间通信要求较高。\n",
" - 比数据并行复杂。\n",
"\n",
"#### **适用场景**\n",
"- 超大模型的高效训练。\n",
"\n",
"#### **具体实现**\n",
"- DeepSpeed 提供的 ZeRO Stage 1/2/3。\n",
"\n",
"---\n",
"\n",
"### **方法对比**\n",
"\n",
"| 并行方法 | 主要优点 | 主要缺点 | 适用场景 |\n",
"|---------------|-------------------------------|-------------------------------|---------------------------|\n",
"| 数据并行 | 简单高效，易实现 | 模型副本占用大量显存 | 模型规模适中，显存足够 |\n",
"| 模型并行 | 支持大模型 | 通信开销大，切分复杂 | 超大模型，显存有限 |\n",
"| 张量并行 | 高效利用显存 | 实现复杂，通信频繁 | 参数规模极大的模型 |\n",
"| 管道并行 | 显存需求降低，适合深模型 | 流水线延迟，数据同步复杂 | 层数多的大型模型 |\n",
"| 混合并行 | 灵活适配超大规模模型 | 配置复杂，依赖框架 | 超大规模模型（如 GPT-3） |\n",
"| ZeRO 并行 | 极大节省显存，占用少 | 通信成本高 | 超大规模模型显存优化 |\n",
"\n",
"---\n",
"\n",
"### **总结**\n",
"- **中等规模模型**：优先使用 **数据并行**。\n",
"- **单卡显存不足**：采用 **模型并行** 或 **张量并行**。\n",
"- **超大规模模型**：使用 **混合并行** 或 DeepSpeed 的 **ZeRO 优化**。\n",
"\n",
"对于现代超大规模模型，通常采用混合并行方法，比如 NVIDIA 的 Megatron-LM 和微软的 DeepSpeed，它们综合了多种并行策略，能够有效利用计算资源并加速训练。如果您有具体的硬件环境或模型需求，可以进一步探讨适合的并行方案！"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06ddaa4d-e04a-41e0-beb5-f04dfaebcd54",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "c0d29667-1e75-46df-8f65-cae27609ee3f",
"metadata": {},
"source": [
"## deepspeed简介\n",
"\n",
"DeepSpeed 是微软开发的开源深度学习优化库，专为大规模模型训练和推理设计，能够显著提升训练速度、降低显存占用，并支持分布式计算。以下是 DeepSpeed 的关键特点和功能：\n",
"\n",
"---\n",
"\n",
"### **1. 核心特点**\n",
"\n",
"#### **（1）高效分布式训练**\n",
"DeepSpeed 提供先进的分布式技术（如 ZeRO 优化器），支持数百亿甚至上万亿参数的模型训练，同时降低单设备显存需求。\n",
"\n",
"#### **（2）显存优化**\n",
"通过显存分片（ZeRO）、梯度累积和混合精度训练，DeepSpeed 能够在有限显存的情况下训练大模型。\n",
"\n",
"#### **（3）性能提升**\n",
"DeepSpeed 优化了通信和计算过程，可提升多 GPU 分布式训练效率。\n",
"\n",
"#### **（4）灵活性**\n",
"与 PyTorch 无缝集成，并兼容 Hugging Face `transformers` 和其他主流深度学习库。\n",
"\n",
"#### **（5）推理优化**\n",
"支持高效推理（如量化和张量并行），适合大模型的生产部署。\n",
"\n",
"---\n",
"\n",
"### **2. 核心技术**\n",
"\n",
"#### **（1）ZeRO 优化器**\n",
"ZeRO（Zero Redundancy Optimizer）是 DeepSpeed 的核心技术之一，分为 3 个阶段：\n",
"- **Stage 1**：分片优化器状态（如动量、方差）。\n",
"- **Stage 2**：分片优化器状态和梯度。\n",
"- **Stage 3**：分片优化器状态、梯度和模型参数，实现全分片优化。\n",
"\n",
"每个阶段都进一步减少显存需求，Stage 3 可支持超大规模模型（如 GPT-3）。\n",
"\n",
"<img src='img/deepspeed.png' width='600px' />\n",
"\n",
"#### **（2）混合精度训练**\n",
"通过 FP16 或 BF16（半精度浮点数）计算，显著减少显存占用并提升计算效率。\n",
"\n",
"#### **（3）数据并行与模型并行**\n",
"- 数据并行：将数据划分到多个设备，每个设备计算部分梯度。\n",
"- 模型并行：将模型的不同部分分配到多个设备。\n",
"- 张量并行：将张量运算分解到多个 GPU 上。\n",
"\n",
"#### **（4）梯度累积**\n",
"支持更大的有效批量大小，适合显存受限的设备。\n",
"\n",
"#### **（5）推理优化**\n",
"- 推理阶段的显存优化和加速技术。\n",
"- 量化推理，减少模型大小和运行时开销。\n",
"\n",
"---\n",
"\n",
"### **3. 适用场景**\n",
"\n",
"#### **（1）大规模模型训练**\n",
"适合训练数十亿或上万亿参数的模型，如 GPT-3、BERT、T5 等。\n",
"\n",
"#### **（2）分布式训练**\n",
"支持单机多卡、多机多卡分布式训练，能高效利用多 GPU 环境。\n",
"\n",
"#### **（3）显存受限的模型微调**\n",
"通过显存优化技术，能在较小显存设备（如 16GB GPU）上微调大模型。\n",
"\n",
"#### **（4）高效推理**\n",
"适用于大语言模型的生产部署，支持推理加速和量化。\n",
"\n",
"---\n",
"\n",
"### **4. 优势与局限性**\n",
"\n",
"#### **优势**\n",
"1. 显存需求显著降低，适合超大规模模型训练。\n",
"2. 支持多种分布式模式，扩展性强。\n",
"3. 与 PyTorch 和 Hugging Face 无缝集成。\n",
"4. 推理优化技术降低部署成本。\n",
"\n",
"#### **局限性**\n",
"1. 配置和调优可能较为复杂。\n",
"2. 对小规模模型或数据集的性能提升有限。\n",
"\n",
"---\n",
"\n",
"### **5. 安装与基本用法**\n",
"\n",
"#### **安装**\n",
"```bash\n",
"pip install deepspeed\n",
"```\n",
"\n",
"#### **基本用法**\n",
"DeepSpeed 通过配置文件启用特性，例如 ZeRO 优化器：\n",
"```python\n",
"from transformers import GPT2LMHeadModel, TrainingArguments, Trainer\n",
"import deepspeed\n",
"\n",
"# 配置 DeepSpeed\n",
"deepspeed_config = {\n",
" \"train_batch_size\": 64,\n",
" \"gradient_accumulation_steps\": 8,\n",
" \"fp16\": {\n",
" \"enabled\": True\n",
" },\n",
" \"zero_optimization\": {\n",
" \"stage\": 2,\n",
" \"overlap_comm\": True\n",
" }\n",
"}\n",
"\n",
"# 保存配置文件\n",
"import json\n",
"with open(\"deepspeed_config.json\", \"w\") as f:\n",
" json.dump(deepspeed_config, f)\n",
"\n",
"# 集成到 Hugging Face Trainer\n",
"training_args = TrainingArguments(\n",
" output_dir=\"./results\",\n",
" per_device_train_batch_size=4,\n",
" num_train_epochs=3,\n",
" learning_rate=5e-5,\n",
" fp16=True,\n",
" deepspeed=\"./deepspeed_config.json\" # DeepSpeed 配置文件\n",
")\n",
"\n",
"trainer = Trainer(\n",
" model=GPT2LMHeadModel.from_pretrained(\"gpt2\"),\n",
" args=training_args,\n",
" train_dataset=train_dataset,\n",
" eval_dataset=eval_dataset\n",
")\n",
"\n",
"trainer.train()\n",
"```\n",
"\n",
"---\n",
"\n",
"### **6. 总结**\n",
"\n",
"DeepSpeed 是大模型训练的强力工具，特别是在多 GPU 环境下，其显存优化和分布式训练技术能显著提升训练效率。适用于以下场景：\n",
"- 超大规模模型的训练和微调。\n",
"- 多机多卡环境的分布式训练。\n",
"- 高效推理部署。\n",
"\n",
"如果需要进一步优化模型训练或部署性能，DeepSpeed 是值得尝试的工具！"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a5372798-ced3-420c-b853-badd3ff05dc1",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "cd848439-bac8-46b2-9a0f-59ae7c343954",
"metadata": {},
"source": [
"## deepspeed具体设置\n",
"\n",
"\n",
"是的，DeepSpeed 支持多种并行策略，包括 **数据并行**、**模型并行** 和 **张量并行**，并且可以通过其配置文件灵活地设置这些并行模式。\n",
"\n",
"---\n",
"\n",
"### **1. 数据并行**\n",
"\n",
"#### **原理**\n",
"在数据并行中，DeepSpeed 将数据批次划分到多个 GPU，每个 GPU 上都有模型的完整副本，计算独立的梯度。最终通过 `AllReduce` 操作同步梯度并更新模型参数。\n",
"\n",
"#### **如何设置**\n",
"DeepSpeed 默认支持数据并行，启用 `zero_optimization` 后会自动结合 ZeRO 优化器进行分片数据并行：\n",
"```json\n",
"{\n",
" \"train_batch_size\": 64,\n",
" \"gradient_accumulation_steps\": 8,\n",
" \"fp16\": {\n",
" \"enabled\": true\n",
" },\n",
" \"zero_optimization\": {\n",
" \"stage\": 1\n",
" }\n",
"}\n",
"```\n",
"\n",
"---\n",
"\n",
"### **2. 模型并行**\n",
"\n",
"#### **原理**\n",
"模型并行将模型的不同部分（如 Transformer 层或权重张量）分布到多个 GPU。DeepSpeed 本身不直接实现模型并行，但可以与模型并行框架（如 NVIDIA Megatron-LM）集成。\n",
"\n",
"#### **如何设置**\n",
"如果使用模型并行（如层级分割）：\n",
"1. 使用 DeepSpeed 的 Pipeline Parallelism：\n",
" ```json\n",
" {\n",
" \"train_batch_size\": 64,\n",
" \"pipeline_parallel_size\": 2, # 设置流水线并行 GPU 数量\n",
" \"fp16\": {\n",
" \"enabled\": true\n",
" },\n",
" \"zero_optimization\": {\n",
" \"stage\": 1\n",
" }\n",
" }\n",
" ```\n",
"\n",
"2. 与 NVIDIA Megatron-LM 集成：\n",
" 在代码中使用 Megatron-LM 的模型并行支持，然后结合 DeepSpeed：\n",
" ```python\n",
" from megatron import get_model_parallel_world_size\n",
" import deepspeed\n",
"\n",
" model = MyModel(...)\n",
" model = deepspeed.initialize(\n",
" model=model,\n",
" model_parallel_size=get_model_parallel_world_size(),\n",
" config=\"./deepspeed_config.json\"\n",
" )\n",
" ```\n",
"\n",
"---\n",
"\n",
"### **3. 张量并行**\n",
"\n",
"#### **原理**\n",
"张量并行将模型参数张量（如权重矩阵）分片到多个 GPU，并通过通信协作完成计算。DeepSpeed 提供了张量并行的支持（在 ZeRO Stage 3 中），或者通过集成 Megatron-LM 实现。\n",
"\n",
"#### **如何设置**\n",
"1. **使用 ZeRO Stage 3**：\n",
" ZeRO Stage 3 会分片模型参数和优化器状态，类似于张量并行的效果：\n",
" ```json\n",
" {\n",
" \"train_batch_size\": 64,\n",
" \"gradient_accumulation_steps\": 8,\n",
" \"fp16\": {\n",
" \"enabled\": true\n",
" },\n",
" \"zero_optimization\": {\n",
" \"stage\": 3,\n",
" \"offload_optimizer\": {\n",
" \"device\": \"cpu\",\n",
" \"pin_memory\": true\n",
" },\n",
" \"offload_param\": {\n",
" \"device\": \"cpu\",\n",
" \"pin_memory\": true\n",
" }\n",
" }\n",
" }\n",
" ```\n",
"\n",
"2. **集成 Megatron-LM**：\n",
" 如果需要更复杂的张量并行方案（如矩阵切分），可以通过 Megatron-LM 实现，然后与 DeepSpeed 集成。\n",
"\n",
"---\n",
"\n",
"### **4. 混合并行**\n",
"\n",
"#### **原理**\n",
"混合并行结合了数据并行、模型并行和张量并行。DeepSpeed 提供了对这些模式的集成支持，允许您灵活配置。\n",
"\n",
"#### **如何设置**\n",
"结合数据并行和流水线并行：\n",
"```json\n",
"{\n",
" \"train_batch_size\": 64,\n",
" \"gradient_accumulation_steps\": 8,\n",
" \"fp16\": {\n",
" \"enabled\": true\n",
" },\n",
" \"pipeline_parallel_size\": 2, # 流水线并行\n",
" \"zero_optimization\": {\n",
" \"stage\": 2\n",
" }\n",
"}\n",
"```\n",
"\n",
"与张量并行结合：\n",
"1. 在代码中配置张量并行：\n",
" ```python\n",
" from megatron import get_tensor_parallel_world_size\n",
" model = MyModel(...)\n",
" model = deepspeed.initialize(\n",
" model=model,\n",
" tensor_parallel_size=get_tensor_parallel_world_size(),\n",
" config=\"./deepspeed_config.json\"\n",
" )\n",
" ```\n",
"\n",
"2. DeepSpeed 配置文件中启用 ZeRO Stage 3。\n",
"\n",
"---\n",
"\n",
"### **5. 选择并行策略**\n",
"\n",
"| 并行模式 | **支持方式** | **适用场景** |\n",
"|---------------|------------------------------------------|-----------------------------------------|\n",
"| 数据并行 | 默认支持，结合 ZeRO 优化器 | 模型参数较小，显存压力不大的场景 |\n",
"| 模型并行 | 使用 Pipeline Parallelism 或集成 Megatron-LM | 模型参数非常大，单 GPU 无法容纳的场景 |\n",
"| 张量并行 | ZeRO Stage 3 或集成 Megatron-LM | 参数矩阵非常大，需要分片计算的场景 |\n",
"| 混合并行 | 结合数据并行、模型并行和张量并行 | 超大规模模型（如 GPT-3）训练 |\n",
"\n",
"---\n",
"\n",
"### **6. 示例代码**\n",
"\n",
"以下是集成 ZeRO 和 Pipeline Parallelism 的完整示例：\n",
"```python\n",
"import deepspeed\n",
"from transformers import GPT2LMHeadModel, TrainingArguments, Trainer\n",
"from datasets import load_dataset\n",
"\n",
"# 加载数据\n",
"dataset = load_dataset(\"wikitext\", \"wikitext-2-raw-v1\", split=\"train\")\n",
"\n",
"# 加载模型\n",
"model = GPT2LMHeadModel.from_pretrained(\"gpt2\")\n",
"\n",
"# 配置 DeepSpeed\n",
"deepspeed_config = {\n",
" \"train_batch_size\": 64,\n",
" \"gradient_accumulation_steps\": 8,\n",
" \"pipeline_parallel_size\": 2, # 流水线并行\n",
" \"fp16\": {\n",
" \"enabled\": True\n",
" },\n",
" \"zero_optimization\": {\n",
" \"stage\": 2\n",
" }\n",
"}\n",
"\n",
"# 保存配置文件\n",
"import json\n",
"with open(\"deepspeed_config.json\", \"w\") as f:\n",
" json.dump(deepspeed_config, f)\n",
"\n",
"# 训练参数\n",
"training_args = TrainingArguments(\n",
" output_dir=\"./results\",\n",
" per_device_train_batch_size=4,\n",
" num_train_epochs=3,\n",
" deepspeed=\"./deepspeed_config.json\", # 指定 DeepSpeed 配置文件\n",
")\n",
"\n",
"# 初始化 Trainer\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=dataset\n",
")\n",
"\n",
"# 开始训练\n",
"trainer.train()\n",
"```\n",
"\n",
"---\n",
"\n",
"### **总结**\n",
"\n",
"- **数据并行**：默认支持，结合 ZeRO 进行优化。\n",
"- **模型并行**：使用 Pipeline Parallelism 或与 Megatron-LM 集成。\n",
"- **张量并行**：通过 ZeRO Stage 3 或 Megatron-LM 实现。\n",
"- **混合并行**：灵活结合多种并行方法，用于超大规模模型。\n",
"\n",
"DeepSpeed 的配置高度灵活，可以根据模型大小、显存限制和硬件条件选择适合的并行策略。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a8e6de4c-adc1-4a1b-840a-c8542b4ed783",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "3383c2d7-91a9-4940-b3b2-698fb7d9dbb7",
"metadata": {},
"source": [
"## 使用gpt2+deepspeed训练"
]
},
{
"cell_type": "markdown",
"id": "ab2812bc-f743-4f18-b49c-972781484dc6",
"metadata": {},
"source": [
"## gpt2的训练\n",
"\n",
"```\n",
"#一般方式训练gpt2\n",
"python pretain_gpt2.py\n",
"\n",
"\n",
"#deepspeed训练gpt2, 只多一行代码\n",
"torchrun --nproc_per_node=6 deepspeed_pretrain_gpt2.py\n",
"\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9cb60dc2-4cec-492d-836b-67694829acf2",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/5-peft-intro-checkpoint.ipynb
+987
View File
@@ -0,0 +1,987 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "963e9ae0-ac68-44be-8c7d-fb9842784362",
"metadata": {},
"source": [
"# 4.5 peft简介"
]
},
{
"cell_type": "markdown",
"id": "f4288594-c676-4369-aca1-730446f293d7",
"metadata": {},
"source": [
"## peft"
]
},
{
"cell_type": "markdown",
"id": "182b82c4-d484-4c15-a600-03c3b51367ec",
"metadata": {},
"source": [
"**PEFT**（Parameter-Efficient Fine-Tuning，参数高效微调）是一种优化技术，旨在以最小的参数更新实现对大规模预训练模型（如 GPT、BERT 等）的微调。PEFT 技术通过减少微调所需的参数量，显著降低了存储和计算开销，同时保留模型的性能，特别适合资源受限的场景和领域特定任务的定制化。\n",
"\n",
"---\n",
"\n",
"### **1. 核心思想**\n",
"传统的微调方式需要更新整个预训练模型的所有参数，PEFT 技术通过只调整少量的参数（如特定层或额外添加的小型模块）实现微调目标，大幅减少了训练开销和存储需求。\n",
"\n",
"---\n",
"\n",
"### **2. 常见的 PEFT 方法**\n",
"\n",
"#### **（1）Adapter 模型**\n",
"- 在每一层 Transformer 的输出中插入小型适配器模块，仅训练适配器模块的参数。\n",
"- 原始模型参数保持冻结不变。\n",
"- 优点：适配器模块参数量小，能适应不同任务。\n",
"\n",
"示例方法：\n",
"- **AdapterFusion**\n",
"- **MAD-X**\n",
"\n",
"---\n",
"\n",
"#### **（2）Prefix Tuning**\n",
"- 在 Transformer 的输入前添加一组可学习的前缀向量，这些前缀与模型的注意力机制交互。\n",
"- 只调整前缀向量的参数，而不更新原始模型。\n",
"- 优点：对生成任务效果显著，参数量进一步减少。\n",
"\n",
"---\n",
"\n",
"#### **（3）LoRA（Low-Rank Adaptation）**\n",
"- 将预训练模型中的部分权重分解为两个低秩矩阵，仅调整这些低秩矩阵的参数。\n",
"- 原始权重保持冻结状态。\n",
"- 优点：参数量极小，计算高效。\n",
" \n",
"---\n",
"\n",
"#### **（4）Prompt Tuning**\n",
"- 在输入文本中添加可学习的提示（Prompt）。\n",
"- 适合 NLP 任务中的文本生成、分类等。\n",
"- 优点：实现简单，易于集成到现有框架。\n",
"\n",
"---\n",
"\n",
"### **3. PEFT 的优势**\n",
"\n",
"1. **显著减少参数更新量**：\n",
" - 微调传统的大模型（如 GPT-3）需要更新数百亿参数，而 PEFT 仅需更新百万级别甚至更少的参数。\n",
"\n",
"2. **高效存储**：\n",
" - 每个任务的微调结果只需存储少量额外参数，而不是整个模型。\n",
"\n",
"3. **适用多任务**：\n",
" - 同一预训练模型可以通过不同的 PEFT 模块适配多个任务，无需重新训练。\n",
"\n",
"4. **降低计算开销**：\n",
" - 训练所需的内存和计算显著减少，适合资源有限的环境。\n",
"\n",
"---\n",
"\n",
"### **4. 应用场景**\n",
"\n",
"1. **领域特定任务**：\n",
" - 医疗、法律、金融等领域微调预训练模型。\n",
"\n",
"2. **多任务学习**：\n",
" - 适配多个任务，复用同一模型的预训练权重。\n",
"\n",
"3. **资源受限场景**：\n",
" - 移动设备、边缘设备上的模型部署。\n",
"\n",
"---\n",
"\n",
"### **5. Hugging Face PEFT 库**\n",
"\n",
"Hugging Face 提供了专门的 PEFT 库，支持多种参数高效微调技术：\n",
"- **安装**：\n",
" ```bash\n",
" pip install peft\n",
" ```\n",
"- **使用 LoRA 微调示例**：\n",
" ```python\n",
" from transformers import AutoModelForCausalLM, AutoTokenizer\n",
" from peft import LoraConfig, get_peft_model, TaskType\n",
"\n",
" # 加载模型和分词器\n",
" model_name = \"gpt2\"\n",
" model = AutoModelForCausalLM.from_pretrained(model_name)\n",
" tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
"\n",
" # 配置 LoRA\n",
" lora_config = LoraConfig(\n",
" task_type=TaskType.CAUSAL_LM,\n",
" r=8,\n",
" lora_alpha=32,\n",
" target_modules=[\"q_proj\", \"v_proj\"],\n",
" lora_dropout=0.1,\n",
" bias=\"none\"\n",
" )\n",
"\n",
" # 使用 LoRA 微调模型\n",
" model = get_peft_model(model, lora_config)\n",
" model.print_trainable_parameters()\n",
"\n",
" # 微调代码...\n",
" ```\n",
"\n",
"---\n",
"\n",
"### **6. PEFT 的局限性**\n",
"1. **特定任务限制**：\n",
" - 在一些复杂任务中，PEFT 方法可能不如全量微调效果好。\n",
"\n",
"2. **需要设计合适的模块**：\n",
" - 不同任务需要选择和设计合适的 PEFT 技术。\n",
"\n",
"3. **与模型架构相关**：\n",
" - PEFT 技术可能需要对模型架构进行一定程度的修改。\n",
"\n",
"---\n",
"\n",
"### **7. 小结**\n",
"PEFT 是一个极具潜力的技术，特别适合在有限资源下对大模型进行微调。它在许多领域和任务中已显示出良好的效果，例如 LoRA 和 Adapter 模型已经成为高效微调的主流方法。\n",
"\n",
"如果您需要实现高效微调，可以结合 Hugging Face 的 PEFT 库快速上手。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a70b2631-c9b9-49da-96c6-6760c63040ac",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "7b47ddf3-85c9-4dd8-bbbb-34fc3bd6aa1b",
"metadata": {},
"source": [
"## GPT2使用peft样例"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5aa3d240-44e1-4811-8f61-d6ff2500a798",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "markdown",
"id": "17bdb69d-3f0f-465e-bd60-2047a088e264",
"metadata": {},
"source": [
"如果您不确定模型中有哪些模块可以微调，可以打印模型结构："
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "41a0c049-9134-4d89-aad0-1aa2241a9fca",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4becc479adbc472bb7672d49da16aafd",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"generation_config.json: 0%| | 0.00/124 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"transformer\n",
"transformer.wte\n",
"transformer.wpe\n",
"transformer.drop\n",
"transformer.h\n",
"transformer.h.0\n",
"transformer.h.0.ln_1\n",
"transformer.h.0.attn\n",
"transformer.h.0.attn.c_attn\n",
"transformer.h.0.attn.c_proj\n",
"transformer.h.0.attn.attn_dropout\n",
"transformer.h.0.attn.resid_dropout\n",
"transformer.h.0.ln_2\n",
"transformer.h.0.mlp\n",
"transformer.h.0.mlp.c_fc\n",
"transformer.h.0.mlp.c_proj\n",
"transformer.h.0.mlp.act\n",
"transformer.h.0.mlp.dropout\n",
"transformer.h.1\n",
"transformer.h.1.ln_1\n",
"transformer.h.1.attn\n",
"transformer.h.1.attn.c_attn\n",
"transformer.h.1.attn.c_proj\n",
"transformer.h.1.attn.attn_dropout\n",
"transformer.h.1.attn.resid_dropout\n",
"transformer.h.1.ln_2\n",
"transformer.h.1.mlp\n",
"transformer.h.1.mlp.c_fc\n",
"transformer.h.1.mlp.c_proj\n",
"transformer.h.1.mlp.act\n",
"transformer.h.1.mlp.dropout\n",
"transformer.h.2\n",
"transformer.h.2.ln_1\n",
"transformer.h.2.attn\n",
"transformer.h.2.attn.c_attn\n",
"transformer.h.2.attn.c_proj\n",
"transformer.h.2.attn.attn_dropout\n",
"transformer.h.2.attn.resid_dropout\n",
"transformer.h.2.ln_2\n",
"transformer.h.2.mlp\n",
"transformer.h.2.mlp.c_fc\n",
"transformer.h.2.mlp.c_proj\n",
"transformer.h.2.mlp.act\n",
"transformer.h.2.mlp.dropout\n",
"transformer.h.3\n",
"transformer.h.3.ln_1\n",
"transformer.h.3.attn\n",
"transformer.h.3.attn.c_attn\n",
"transformer.h.3.attn.c_proj\n",
"transformer.h.3.attn.attn_dropout\n",
"transformer.h.3.attn.resid_dropout\n",
"transformer.h.3.ln_2\n",
"transformer.h.3.mlp\n",
"transformer.h.3.mlp.c_fc\n",
"transformer.h.3.mlp.c_proj\n",
"transformer.h.3.mlp.act\n",
"transformer.h.3.mlp.dropout\n",
"transformer.h.4\n",
"transformer.h.4.ln_1\n",
"transformer.h.4.attn\n",
"transformer.h.4.attn.c_attn\n",
"transformer.h.4.attn.c_proj\n",
"transformer.h.4.attn.attn_dropout\n",
"transformer.h.4.attn.resid_dropout\n",
"transformer.h.4.ln_2\n",
"transformer.h.4.mlp\n",
"transformer.h.4.mlp.c_fc\n",
"transformer.h.4.mlp.c_proj\n",
"transformer.h.4.mlp.act\n",
"transformer.h.4.mlp.dropout\n",
"transformer.h.5\n",
"transformer.h.5.ln_1\n",
"transformer.h.5.attn\n",
"transformer.h.5.attn.c_attn\n",
"transformer.h.5.attn.c_proj\n",
"transformer.h.5.attn.attn_dropout\n",
"transformer.h.5.attn.resid_dropout\n",
"transformer.h.5.ln_2\n",
"transformer.h.5.mlp\n",
"transformer.h.5.mlp.c_fc\n",
"transformer.h.5.mlp.c_proj\n",
"transformer.h.5.mlp.act\n",
"transformer.h.5.mlp.dropout\n",
"transformer.h.6\n",
"transformer.h.6.ln_1\n",
"transformer.h.6.attn\n",
"transformer.h.6.attn.c_attn\n",
"transformer.h.6.attn.c_proj\n",
"transformer.h.6.attn.attn_dropout\n",
"transformer.h.6.attn.resid_dropout\n",
"transformer.h.6.ln_2\n",
"transformer.h.6.mlp\n",
"transformer.h.6.mlp.c_fc\n",
"transformer.h.6.mlp.c_proj\n",
"transformer.h.6.mlp.act\n",
"transformer.h.6.mlp.dropout\n",
"transformer.h.7\n",
"transformer.h.7.ln_1\n",
"transformer.h.7.attn\n",
"transformer.h.7.attn.c_attn\n",
"transformer.h.7.attn.c_proj\n",
"transformer.h.7.attn.attn_dropout\n",
"transformer.h.7.attn.resid_dropout\n",
"transformer.h.7.ln_2\n",
"transformer.h.7.mlp\n",
"transformer.h.7.mlp.c_fc\n",
"transformer.h.7.mlp.c_proj\n",
"transformer.h.7.mlp.act\n",
"transformer.h.7.mlp.dropout\n",
"transformer.h.8\n",
"transformer.h.8.ln_1\n",
"transformer.h.8.attn\n",
"transformer.h.8.attn.c_attn\n",
"transformer.h.8.attn.c_proj\n",
"transformer.h.8.attn.attn_dropout\n",
"transformer.h.8.attn.resid_dropout\n",
"transformer.h.8.ln_2\n",
"transformer.h.8.mlp\n",
"transformer.h.8.mlp.c_fc\n",
"transformer.h.8.mlp.c_proj\n",
"transformer.h.8.mlp.act\n",
"transformer.h.8.mlp.dropout\n",
"transformer.h.9\n",
"transformer.h.9.ln_1\n",
"transformer.h.9.attn\n",
"transformer.h.9.attn.c_attn\n",
"transformer.h.9.attn.c_proj\n",
"transformer.h.9.attn.attn_dropout\n",
"transformer.h.9.attn.resid_dropout\n",
"transformer.h.9.ln_2\n",
"transformer.h.9.mlp\n",
"transformer.h.9.mlp.c_fc\n",
"transformer.h.9.mlp.c_proj\n",
"transformer.h.9.mlp.act\n",
"transformer.h.9.mlp.dropout\n",
"transformer.h.10\n",
"transformer.h.10.ln_1\n",
"transformer.h.10.attn\n",
"transformer.h.10.attn.c_attn\n",
"transformer.h.10.attn.c_proj\n",
"transformer.h.10.attn.attn_dropout\n",
"transformer.h.10.attn.resid_dropout\n",
"transformer.h.10.ln_2\n",
"transformer.h.10.mlp\n",
"transformer.h.10.mlp.c_fc\n",
"transformer.h.10.mlp.c_proj\n",
"transformer.h.10.mlp.act\n",
"transformer.h.10.mlp.dropout\n",
"transformer.h.11\n",
"transformer.h.11.ln_1\n",
"transformer.h.11.attn\n",
"transformer.h.11.attn.c_attn\n",
"transformer.h.11.attn.c_proj\n",
"transformer.h.11.attn.attn_dropout\n",
"transformer.h.11.attn.resid_dropout\n",
"transformer.h.11.ln_2\n",
"transformer.h.11.mlp\n",
"transformer.h.11.mlp.c_fc\n",
"transformer.h.11.mlp.c_proj\n",
"transformer.h.11.mlp.act\n",
"transformer.h.11.mlp.dropout\n",
"transformer.ln_f\n",
"lm_head\n"
]
}
],
"source": [
"from transformers import AutoModelForCausalLM\n",
"\n",
"model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n",
"\n",
"# 打印所有模块名称\n",
"for name, module in model.named_modules():\n",
" print(name)"
]
},
{
"cell_type": "markdown",
"id": "0add2f79-f35c-4638-80bb-0d8a87a9b6a7",
"metadata": {},
"source": [
"在选择 `target_modules` 时，通常会根据模块的名称选择模型的特定部分，通常使用列表中最后一个点 `.` 后的字段名或整个路径名（如果需要更精确）。以下是对这些模块的详细分析和选择建议：\n",
"\n",
"---\n",
"\n",
"### **1. 分析模块结构**\n",
"\n",
"从列表中可以看出，GPT-2 的模块层次分为以下几类：\n",
"\n",
"1. **Embedding 层**：\n",
" - `transformer.wte`：词嵌入层（Word Token Embeddings）。\n",
" - `transformer.wpe`：位置嵌入层（Position Embeddings）。\n",
"\n",
"2. **Transformer 编码器层**：\n",
" - 每层编号为 `transformer.h.<层号>`（共 12 层）。\n",
" - 每层中包含：\n",
" - **层归一化**：\n",
" - `transformer.h.<层号>.ln_1`：第一层归一化。\n",
" - `transformer.h.<层号>.ln_2`：第二层归一化。\n",
" - **自注意力模块**：\n",
" - `transformer.h.<层号>.attn.c_attn`：注意力模块的 Query、Key 和 Value 投影。\n",
" - `transformer.h.<层号>.attn.c_proj`：注意力的输出投影。\n",
" - `transformer.h.<层号>.attn.attn_dropout`：注意力的 Dropout。\n",
" - `transformer.h.<层号>.attn.resid_dropout`：残差的 Dropout。\n",
" - **前馈网络模块（MLP）**：\n",
" - `transformer.h.<层号>.mlp.c_fc`：MLP 的第一层全连接。\n",
" - `transformer.h.<层号>.mlp.c_proj`：MLP 的第二层全连接（输出投影）。\n",
" - `transformer.h.<层号>.mlp.act`：激活函数（如 GELU）。\n",
" - `transformer.h.<层号>.mlp.dropout`：MLP 的 Dropout。\n",
"\n",
"3. **最终层**：\n",
" - `transformer.ln_f`：最终层归一化（LayerNorm）。\n",
" - `lm_head`：语言建模头，用于生成预测的 token 分布。\n",
"\n",
"---\n",
"\n",
"### **2. 如何选择 `target_modules`**\n",
"\n",
"#### **（1）常见目标模块**\n",
"- `transformer.h.<层号>.attn.c_attn`：对自注意力模块的 Query、Key 和 Value 投影层微调。\n",
"- `transformer.h.<层号>.attn.c_proj`：对注意力输出的投影层微调。\n",
"- `transformer.h.<层号>.mlp.c_fc`：对前馈网络的输入全连接层微调。\n",
"- `transformer.h.<层号>.mlp.c_proj`：对前馈网络的输出投影层微调。\n",
"\n",
"#### **（2）推荐设置**\n",
"- **文本生成任务**：\n",
" ```python\n",
" target_modules = [\"transformer.h.*.attn.c_attn\", \"transformer.h.*.attn.c_proj\"]\n",
" ```\n",
" 解释：\n",
" - `*.attn.c_attn`：调整 Query、Key、Value 的生成。\n",
" - `*.attn.c_proj`：调整注意力输出。\n",
"\n",
"- **文本分类任务**：\n",
" ```python\n",
" target_modules = [\"transformer.h.*.attn.c_attn\"]\n",
" ```\n",
" 解释：\n",
" - 微调自注意力模块最重要的部分即可。\n",
"\n",
"- **特定任务需要更细粒度控制**：\n",
" - 仅微调某几层：\n",
" ```python\n",
" target_modules = [\"transformer.h.0.attn.c_attn\", \"transformer.h.0.mlp.c_fc\"]\n",
" ```\n",
"\n",
"#### **（3）通配符选择**\n",
"使用 `*` 通配符可以指定所有层的某些模块：\n",
"- `transformer.h.*.attn.c_attn`：所有层的 Query、Key 和 Value 投影。\n",
"- `transformer.h.*.mlp.*`：所有层的 MLP 模块。\n",
"\n",
"---\n",
"\n",
"### **3. 示例：指定多个模块**\n",
"\n",
"```python\n",
"lora_config = LoraConfig(\n",
" task_type=TaskType.CAUSAL_LM,\n",
" r=8,\n",
" lora_alpha=32,\n",
" target_modules=[\n",
" \"transformer.h.*.attn.c_attn\",\n",
" \"transformer.h.*.mlp.c_fc\"\n",
" ],\n",
" lora_dropout=0.1,\n",
" bias=\"none\"\n",
")\n",
"```\n",
"\n",
"- 这表示对所有层的 `attn.c_attn` 和 `mlp.c_fc` 模块进行 LoRA 微调。\n",
"\n",
"---\n",
"\n",
"### **4. 小提示：如何确定适合的模块**\n",
"\n",
"1. **任务相关性**：\n",
" - 文本生成：优先选择自注意力模块（如 `c_attn`）。\n",
" - 文本分类：通常需要全局语义表示，选择 `attn.c_attn` 或 `mlp.c_fc`。\n",
"\n",
"2. **性能与资源平衡**：\n",
" - 如果显存有限，可以只微调部分层。例如，仅选择浅层和深层的模块：\n",
" ```python\n",
" target_modules = [\"transformer.h.0.attn.c_attn\", \"transformer.h.11.attn.c_attn\"]\n",
" ```\n",
"\n",
"3. **打印模块名称以调试**：\n",
" - 确保选择的 `target_modules` 在模型中实际存在：\n",
" ```python\n",
" for name, _ in model.named_modules():\n",
" if \"c_attn\" in name:\n",
" print(name)\n",
" ```\n",
"\n",
"---\n",
"\n",
"### **建议**\n",
"- 一般情况下，`c_attn` 和 `c_proj` 是首选模块。\n",
"- 使用 `transformer.h.*` 通配符可以轻松指定多层。\n",
"- 根据任务需求和资源限制灵活调整目标模块，以实现最佳性能和效率。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14f20171-0719-4dfa-b888-147b657ebff4",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "b4e7bff2-2a4f-4a1d-9cb1-dd02aead2f85",
"metadata": {},
"source": [
"## LoraConfig具体配置"
]
},
{
"cell_type": "markdown",
"id": "10c99eb9-8007-4297-972e-7be71768c9c3",
"metadata": {},
"source": [
"以下是对 `LoraConfig` 配置的更详细解释，特别是如何设置微调哪些参数、冻结哪些参数，以及一般如何选择这些设置：\n",
"\n",
"---\n",
"\n",
"### **1. `LoraConfig` 参数解析**\n",
"\n",
"```python\n",
"lora_config = LoraConfig(\n",
" task_type=TaskType.SEQ_CLS, # 序列分类任务\n",
" r=8, # 降低矩阵秩\n",
" lora_alpha=32, # LoRA 的 alpha 超参数\n",
" target_modules=[\"c_attn\"], # GPT-2 中的自注意力模块\n",
" lora_dropout=0.1, # dropout 概率\n",
" bias=\"none\", # 是否微调偏置参数\n",
")\n",
"```\n",
"\n",
"#### **（1）`task_type`**\n",
"- 定义任务类型，用于指导 PEFT 的具体行为。\n",
"- **常见选项**：\n",
" - `TaskType.CAUSAL_LM`：自回归语言建模（如 GPT 系列模型）。\n",
" - `TaskType.SEQ_CLS`：序列分类（如情感分析）。\n",
" - `TaskType.TOKEN_CLS`：标注任务（如命名实体识别）。\n",
" - `TaskType.SEQ_2_SEQ_LM`：序列到序列任务（如翻译、摘要）。\n",
"\n",
"**当前设置**：\n",
"- `TaskType.SEQ_CLS` 表示目标是文本分类任务。\n",
"\n",
"---\n",
"\n",
"#### **（2）`r`**\n",
"- 表示 LoRA 的 **秩**（rank），是降低矩阵秩的核心参数。\n",
"- LoRA 通过将模型的权重分解为两个低秩矩阵（`A` 和 `B`），只更新这两个矩阵。\n",
"- `r` 的值越大，微调能力越强，但需要的额外参数也越多。\n",
"- **典型范围**：`4` 至 `64`，大多数任务中 `8` 或 `16` 是常用值。\n",
"\n",
"**当前设置**：\n",
"- `r=8` 表示使用低秩分解，并微调 8 维的参数矩阵。\n",
"\n",
"---\n",
"\n",
"#### **（3）`lora_alpha`**\n",
"- 是 LoRA 的一个缩放因子，用于调节两个低秩矩阵的更新速率。\n",
"- **公式**：实际更新 = LoRA 输出 × `lora_alpha / r`\n",
"- **典型范围**：`16` 至 `128`，较大任务中可以选择更高的值。\n",
"\n",
"**当前设置**：\n",
"- `lora_alpha=32`，表示适中幅度的更新速率。\n",
"\n",
"---\n",
"\n",
"#### **（4）`target_modules`**\n",
"- 指定要应用 LoRA 微调的模块。\n",
"- **常见选择**：\n",
" - 对 Transformer 模型中的 **注意力模块**（如 `query`、`key`、`value`）进行微调，因为这些模块对任务性能影响较大。\n",
" - 对 GPT-2，通常选择 `c_attn`（GPT-2 中负责自注意力机制的组合模块）。\n",
"\n",
"**当前设置**：\n",
"- `target_modules=[\"c_attn\"]` 表示只对 GPT-2 的自注意力模块 `c_attn` 应用 LoRA。\n",
"\n",
"---\n",
"\n",
"#### **（5）`lora_dropout`**\n",
"- 表示 LoRA 层的 dropout 概率，用于防止过拟合。\n",
"- **典型范围**：`0.0` 至 `0.1`，视任务复杂性而定。\n",
"\n",
"**当前设置**：\n",
"- `lora_dropout=0.1`，表示有 10% 的概率随机丢弃 LoRA 层的输出。\n",
"\n",
"---\n",
"\n",
"#### **（6）`bias`**\n",
"- 决定是否微调偏置参数。\n",
"- **选项**：\n",
" - `\"none\"`：不微调任何偏置。\n",
" - `\"all\"`：微调所有偏置。\n",
" - `\"lora_only\"`：只微调 LoRA 层的偏置。\n",
"\n",
"**当前设置**：\n",
"- `bias=\"none\"`，表示所有偏置参数保持冻结。\n",
"\n",
"---\n",
"\n",
"### **5. 总结建议**\n",
"- **微调的参数**：优先选择模型中注意力相关模块。\n",
"- **冻结的参数**：大部分参数默认冻结以节省显存。\n",
"- **配置选择**：根据任务复杂性调整 `r` 和 `target_modules`。\n",
"- **推荐起点**：\n",
" - 文本分类：`target_modules=[\"c_attn\"]`, `r=8`, `lora_dropout=0.1`。\n",
" - 文本生成：`target_modules=[\"q_proj\", \"v_proj\"]`, `r=16`, `lora_dropout=0.1`。\n",
"\n",
"通过这些设置，LoRA 可以在参数量极小的情况下实现高效微调，适合各种任务场景。"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "bbc080ba-3ee8-4bc6-afd9-2a3241f1bcda",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "26d9f362-18cc-471f-b208-f29a6933c06a",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at dnagpt/dna_gpt2_v0 and are newly initialized: ['score.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f7e72521368341d38a2b11028715a871",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/5920 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"trainable params: 296,448 || all params: 109,180,416 || trainable%: 0.2715\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/peft/tuners/lora/layer.py:1264: UserWarning: fan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.\n",
" warnings.warn(\n"
]
}
],
"source": [
"from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer\n",
"from peft import LoraConfig, get_peft_model, TaskType\n",
"from datasets import load_dataset\n",
"from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
"from transformers import DataCollatorWithPadding\n",
"\n",
"# **1. 加载模型和分词器**\n",
"model_name = \"dnagpt/dna_gpt2_v0\" # 基础模型\n",
"num_labels = 2 # 二分类任务\n",
"model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)\n",
"tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
"\n",
"tokenizer.pad_token = tokenizer.eos_token\n",
"model.config.pad_token_id = tokenizer.pad_token_id\n",
"\n",
"\n",
"# **2. 定义数据集**\n",
"# 示例数据集：dna_promoter_300\n",
"dataset = load_dataset(\"dnagpt/dna_promoter_300\")['train'].train_test_split(test_size=0.1)\n",
"\n",
"# **3. 数据预处理**\n",
"def preprocess_function(examples):\n",
" examples['label'] = [int(item) for item in examples['label']]\n",
" return tokenizer(\n",
" examples[\"sequence\"], truncation=True, padding=\"max_length\", max_length=128\n",
" )\n",
"\n",
"tokenized_datasets = dataset.map(preprocess_function, batched=True)\n",
"#tokenized_datasets = tokenized_datasets.rename_column(\"label\", \"labels\") # Hugging Face Trainer 要求标签列名为 'labels'\n",
"\n",
"# 4. 创建一个数据收集器，用于动态填充和遮蔽\n",
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
"\n",
"# **4. 划分数据集**\n",
"train_dataset = tokenized_datasets[\"train\"]\n",
"test_dataset = tokenized_datasets[\"test\"]\n",
"\n",
"# **5. 配置 LoRA**\n",
"lora_config = LoraConfig(\n",
" task_type=TaskType.SEQ_CLS, # 序列分类任务\n",
" r=8, # 降低矩阵秩\n",
" lora_alpha=32, # LoRA 的 alpha 超参数\n",
" target_modules=[\"c_attn\"], # GPT-2 中的自注意力模块\n",
" lora_dropout=0.1, # dropout 概率\n",
" bias=\"none\", # 是否微调偏置参数\n",
")\n",
"\n",
"# 使用 LoRA 包装模型\n",
"model = get_peft_model(model, lora_config)\n",
"model.print_trainable_parameters() # 打印可训练的参数信息"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "7da39e7f-db92-483c-888d-19707ab35c5f",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/transformers/training_args.py:1575: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
" warnings.warn(\n",
"/tmp/ipykernel_2399/742597822.py:28: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n",
" trainer = Trainer(\n"
]
},
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='19980' max='19980' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
" [19980/19980 10:13, Epoch 3/3]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Epoch</th>\n",
" <th>Training Loss</th>\n",
" <th>Validation Loss</th>\n",
" <th>Accuracy</th>\n",
" <th>Precision</th>\n",
" <th>Recall</th>\n",
" <th>F1</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>1</td>\n",
" <td>0.299700</td>\n",
" <td>0.325549</td>\n",
" <td>0.897635</td>\n",
" <td>0.908117</td>\n",
" <td>0.885483</td>\n",
" <td>0.896658</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2</td>\n",
" <td>0.304000</td>\n",
" <td>0.290004</td>\n",
" <td>0.904899</td>\n",
" <td>0.889069</td>\n",
" <td>0.925901</td>\n",
" <td>0.907111</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3</td>\n",
" <td>0.310100</td>\n",
" <td>0.289658</td>\n",
" <td>0.906250</td>\n",
" <td>0.892138</td>\n",
" <td>0.924891</td>\n",
" <td>0.908219</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"训练完成，模型已保存至 ./gpt2_lora_text_classification\n"
]
}
],
"source": [
"# **6. 计算指标**\n",
"def compute_metrics(eval_pred):\n",
" predictions, labels = eval_pred\n",
" preds = predictions.argmax(axis=-1)\n",
" precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average=\"binary\")\n",
" acc = accuracy_score(labels, preds)\n",
" return {\"accuracy\": acc, \"precision\": precision, \"recall\": recall, \"f1\": f1}\n",
"\n",
"# **7. 定义训练参数**\n",
"training_args = TrainingArguments(\n",
" output_dir=\"./gpt2_lora_text_classification\", # 模型保存路径\n",
" evaluation_strategy=\"epoch\", # 每个 epoch 评估一次\n",
" save_strategy=\"epoch\", # 每个 epoch 保存一次\n",
" learning_rate=2e-5, # 学习率\n",
" per_device_train_batch_size=8, # 每设备的批量大小\n",
" per_device_eval_batch_size=8, # 每设备评估的批量大小\n",
" num_train_epochs=10, # 训练轮数\n",
" weight_decay=0.01, # 权重衰减\n",
" logging_dir=\"./logs\", # 日志路径\n",
" fp16=True, # 启用混合精度训练\n",
" save_total_limit=2, # 保留最多两个检查点\n",
" load_best_model_at_end=True, # 加载最佳模型\n",
" metric_for_best_model=\"accuracy\", # 根据准确率选择最佳模型\n",
" greater_is_better=True,\n",
")\n",
"\n",
"# **8. 定义 Trainer**\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=train_dataset,\n",
" eval_dataset=test_dataset,\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")\n",
"\n",
"# **9. 开始训练**\n",
"trainer.train()\n",
"\n",
"# **10. 保存模型**\n",
"model.save_pretrained(\"./gpt2_lora_text_classification\")\n",
"tokenizer.save_pretrained(\"./gpt2_lora_text_classification\")\n",
"\n",
"print(\"训练完成，模型已保存至 ./gpt2_lora_text_classification\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49a60fed-3a7d-4608-98b1-b4e313b94dbb",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
"from peft import PeftModel\n",
"\n",
"# 加载分词器\n",
"model_path = \"./gpt2_lora_text_classification\"\n",
"tokenizer = AutoTokenizer.from_pretrained(model_path)\n",
"\n",
"# 加载微调后的 PEFT 模型\n",
"base_model = AutoModelForSequenceClassification.from_pretrained(\"gpt2\", num_labels=2)\n",
"model = PeftModel.from_pretrained(base_model, model_path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3c0d8f02-c3dc-4961-8b3a-50eefc5f9448",
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"def predict(texts, model, tokenizer):\n",
" \"\"\"\n",
" 使用微调后的 PEFT 模型进行推理。\n",
" \n",
" Args:\n",
" texts (list of str): 待分类的文本列表。\n",
" model (PeftModel): 微调后的模型。\n",
" tokenizer (AutoTokenizer): 分词器。\n",
" \n",
" Returns:\n",
" list of dict: 每个文本的预测结果，包括 logits 和预测的类别标签。\n",
" \"\"\"\n",
" # 对输入文本进行分词和编码\n",
" inputs = tokenizer(\n",
" texts,\n",
" padding=True,\n",
" truncation=True,\n",
" max_length=512,\n",
" return_tensors=\"pt\"\n",
" )\n",
" \n",
" # 将输入数据移动到模型的设备上（CPU/GPU）\n",
" inputs = {key: value.to(model.device) for key, value in inputs.items()}\n",
" \n",
" # 模型推理\n",
" model.eval()\n",
" with torch.no_grad():\n",
" outputs = model(**inputs)\n",
" \n",
" # 获取 logits 并计算预测类别\n",
" logits = outputs.logits\n",
" probs = torch.nn.functional.softmax(logits, dim=-1)\n",
" predictions = torch.argmax(probs, dim=-1)\n",
" \n",
" # 返回每个文本的预测结果\n",
" results = [\n",
" {\"text\": text, \"logits\": logit.tolist(), \"predicted_class\": int(pred)}\n",
" for text, logit, pred in zip(texts, logits, predictions)\n",
" ]\n",
" return results\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9c0cfe65-f4f3-4274-a4f4-1ac13725b15a",
"metadata": {},
"outputs": [],
"source": [
"Text: This movie was fantastic! I loved every part of it.\n",
"Predicted Class: 1\n",
"Logits: [-2.345, 3.567]\n",
"\n",
"Text: The plot was terrible and the acting was worse.\n",
"Predicted Class: 0\n",
"Logits: [4.123, -1.234]\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/6-llama-continue-train-checkpoint.ipynb
+717
View File
@@ -0,0 +1,717 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "1e6d4978-4f0f-4268-aa23-d864857bd6c8",
"metadata": {},
"source": [
"# 4.6 基于llama的基因大模型持续预训练"
]
},
{
"cell_type": "markdown",
"id": "2c201732-e736-463c-8446-637bf517479f",
"metadata": {},
"source": [
"LLaMA（**Large Language Model Meta AI**）是由 Meta（Facebook）开发的一系列大型语言模型，专注于提供高性能和高效的大语言模型，面向学术研究和开发社区。LLaMA 系列主要强调训练效率、模型性能和对计算资源的高效利用，是 GPT 系列模型的有力竞争者之一。\n",
"\n",
"---\n",
"\n",
"### **1. LLaMA 模型概述**\n",
"\n",
"#### **1.1 LLaMA 1**\n",
"- **发布**：2023 年 2 月。\n",
"- **模型参数规模**：\n",
" - 7B（70 亿）\n",
" - 13B（130 亿）\n",
" - 33B（330 亿）\n",
" - 65B（650 亿）\n",
"- **特点**：\n",
" - 专注于效率：与 GPT-3 等模型相比，LLaMA 在相同的训练成本下实现了更高的性能。\n",
" - 针对研究开放：提供预训练模型权重供研究使用。\n",
" - 使用高质量的数据：模型训练使用大量从网络中筛选的高质量文本数据，包括维基百科、书籍和其他高质量来源。\n",
"- **性能**：\n",
" - 在许多 NLP 任务中，LLaMA 的性能超过 GPT-3 和其他同类模型。\n",
" - 参数规模较小的版本（如 LLaMA-13B）性能可与 GPT-3（175B 参数）媲美。\n",
"\n",
"#### **1.2 LLaMA 2**\n",
"- **发布**：2023 年 7 月。\n",
"- **改进**：\n",
" - 增强的训练数据：相比 LLaMA 1，使用了更多的高质量数据。\n",
" - 引入微调版本：发布了开箱即用的对话模型（LLaMA 2-Chat）。\n",
" - 更好的开源支持：LLaMA 2 在商业用途上比 LLaMA 1 更加开放。\n",
"- **模型参数规模**：\n",
" - 7B（70 亿）\n",
" - 13B（130 亿）\n",
" - 70B（700 亿）\n",
"- **性能**：\n",
" - LLaMA 2 的性能相比 LLaMA 1 有显著提升。\n",
" - LLaMA 2-Chat 在对话任务中的表现优于许多现有开源模型。\n",
" - 在多个标准基准（如 MMLU）上超过 GPT-4 和 Claude 的开源实现。\n",
"\n",
"---\n",
"\n",
"### **2. LLaMA 的关键技术特点**\n",
"\n",
"#### **2.1 高效的架构设计**\n",
"- 基于 Transformer 架构。\n",
"- 针对训练效率和推理速度进行了优化，适合研究和开发。\n",
"\n",
"#### **2.2 模型压缩**\n",
"- 提供更小的参数规模（如 7B 和 13B），以便在更低的计算资源上运行。\n",
"- 在性能与参数量之间实现了很好的平衡。\n",
"\n",
"#### **2.3 训练数据**\n",
"- 使用从互联网中提取的高质量数据，注重数据清洗和筛选，避免低质量文本对模型的负面影响。\n",
"\n",
"#### **2.4 微调能力**\n",
"- 支持指令微调（Instruction Tuning）和 RLHF（基于人类反馈的强化学习），特别是在 LLaMA 2-Chat 模型中表现优异。\n",
"\n",
"---\n",
"\n",
"### **3. LLaMA 的性能对比**\n",
"\n",
"#### **与 GPT-3 比较**\n",
"- LLaMA 1-13B 参数模型在许多任务上的性能接近 GPT-3-175B。\n",
"- LLaMA 2-70B 在多个任务上超过 GPT-3。\n",
"\n",
"#### **与其他开源模型比较**\n",
"- LLaMA 2 在许多基准测试中优于其他开源模型（如 Falcon 和 MPT）。\n",
"- LLaMA 2-Chat 提供了与 ChatGPT 类似的对话能力，适用于对话任务。\n",
"\n",
"---\n",
"\n",
"### **4. 应用场景**\n",
"\n",
"1. **研究**：\n",
" - 开源权重适合学术研究，推动了对大语言模型的进一步探索。\n",
"\n",
"2. **对话系统**：\n",
" - LLaMA 2-Chat 专为对话任务设计，适合开发智能客服、聊天机器人等应用。\n",
"\n",
"3. **生成任务**：\n",
" - 支持文本生成、补全、摘要等任务。\n",
"\n",
"4. **微调与定制**：\n",
" - 可以基于特定领域数据进行微调，如医学、法律、教育等领域的专用模型。\n",
"\n",
"---\n",
"\n",
"### **5. 开源与获取方式**\n",
"\n",
"#### **1. 开源**\n",
"- LLaMA 1：需要申请权限才能获得模型权重。\n",
"- LLaMA 2：更加开放，允许商业用途，模型和权重可以通过 Meta 的合作平台获取（如 Hugging Face 和 AWS）。\n",
"\n",
"#### **2. 下载与使用**\n",
"使用 Hugging Face 加载模型：\n",
"```python\n",
"from transformers import AutoModelForCausalLM, AutoTokenizer\n",
"\n",
"model_name = \"meta-llama/Llama-2-7b-hf\" # 替换为具体模型\n",
"tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
"model = AutoModelForCausalLM.from_pretrained(model_name)\n",
"\n",
"# 使用模型生成文本\n",
"inputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\")\n",
"outputs = model.generate(**inputs, max_length=50)\n",
"print(tokenizer.decode(outputs[0], skip_special_tokens=True))\n",
"```\n",
"\n",
"---\n",
"\n",
"### **6. 总结**\n",
"\n",
"#### **优势**\n",
"- **高性能**：在多个基准任务上表现出色。\n",
"- **高效训练**：小参数模型能与大模型媲美。\n",
"- **开放性**：LLaMA 2 提供了较为开放的商用许可。\n",
"\n",
"#### **局限**\n",
"- 模型需要高质量数据和强大算力训练，对推理设备也有一定要求。\n",
"\n",
"LLaMA 系列以其高效和开放的特点，为大模型研究和应用带来了强大动力，是当前大语言模型生态的重要组成部分。"
]
},
{
"cell_type": "markdown",
"id": "7fb0d648-f891-47b9-a644-af5263fa9718",
"metadata": {},
"source": [
"---\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "8b3c9ebb-213b-4dc4-a712-5a819fea3197",
"metadata": {},
"source": [
"**大模型的持续预训练**（Continual Pretraining of Large Models）是指在基础预训练模型（如 GPT、BERT 等）的基础上，通过引入新的数据或特定领域的数据继续进行预训练的过程。这一过程旨在让模型在特定场景或任务中表现更好，同时保留其通用能力。\n",
"\n",
"---\n",
"\n",
"### **1. 持续预训练的概念**\n",
"\n",
"持续预训练是一种在通用大模型的预训练基础上，进一步优化和适配模型的方法，主要包括以下两种场景：\n",
"1. **领域适配**：\n",
" - 将预训练模型在特定领域的数据上继续训练，使其对该领域的语料理解更深刻，例如法律、医学、金融等领域。\n",
"2. **性能优化**：\n",
" - 通过引入更多的通用数据或多样化的数据类型，扩展模型的通用能力，提高性能。\n",
"\n",
"---\n",
"\n",
"### **2. 持续预训练的目标**\n",
"\n",
"1. **提升领域性能**：\n",
" - 在特定领域任务上，模型能够更好地理解特定领域的语言模式和知识。\n",
" \n",
"2. **增强模型鲁棒性**：\n",
" - 通过引入新的数据或增强数据多样性，使模型对未见数据表现更稳定。\n",
"\n",
"3. **优化资源利用**：\n",
" - 通过复用已有的大模型权重，只需训练少量额外步骤，避免从零开始重新训练模型。\n",
"\n",
"---\n",
"\n",
"### **3. 持续预训练的步骤**\n",
"\n",
"#### **（1）数据准备**\n",
"- **领域数据**：针对特定领域（如医学、法律、科技）收集高质量语料。\n",
"- **新语料整合**：补充模型未见过的多样化语料。\n",
"- **数据清洗**：确保数据无噪声、语言风格一致。\n",
"\n",
"#### **（2）模型初始化**\n",
"- 使用现有的预训练模型作为初始权重，例如 Hugging Face 提供的 GPT-2 或 BERT 模型。\n",
"\n",
"#### **（3）训练设置**\n",
"- **超参数调整**：\n",
" - 通常使用较小的学习率（例如 `1e-5` 或 `2e-5`）以避免破坏已有的知识。\n",
"- **训练策略**：\n",
" - 冻结部分参数（如嵌入层或前几层）以保留通用能力，仅调整高层或新加入的部分。\n",
"\n",
"#### **（4）评估和验证**\n",
"- 使用领域任务的数据集对模型进行评估，验证其在目标任务中的改进效果。\n",
"\n",
"---\n",
"\n",
"### **4. 持续预训练的常见方法**\n",
"\n",
"#### **（1）全量持续预训练**\n",
"- 对整个模型的参数进行调整。\n",
"- **优点**：适合较大规模的新数据训练，能显著提升领域性能。\n",
"- **缺点**：计算资源需求大，可能导致模型过拟合。\n",
"\n",
"#### **（2）冻结部分参数**\n",
"- 冻结低层参数，仅微调高层。\n",
"- **优点**：保留通用知识，减少计算开销。\n",
"- **缺点**：对领域特定知识的适配可能不足。\n",
"\n",
"#### **（3）参数高效微调（PEFT）**\n",
"- 使用 PEFT 方法（如 LoRA、Adapter）进行预训练：\n",
" - **LoRA**：通过低秩矩阵分解，微调部分关键模块。\n",
" - **Adapter**：在 Transformer 层中插入小型适配模块。\n",
"- **优点**：显著减少需要更新的参数量。\n",
"\n",
"---\n",
"\n",
"### **5. 持续预训练的典型应用**\n",
"\n",
"1. **领域适配**\n",
" - **医学**：将预训练模型在 PubMed 或生物医学数据集上进行持续预训练。\n",
" - **法律**：使用法律文档进一步训练基础模型。\n",
" - **金融**：通过金融新闻、报告语料提升模型在金融领域的表现。\n",
"\n",
"2. **多语言扩展**\n",
" - 引入多语言语料，扩展模型的多语言能力。\n",
"\n",
"3. **数据更新**\n",
" - 持续加入新数据（如时事新闻）以适配最新语言模式。\n",
"\n",
"4. **特殊任务优化**\n",
" - 针对特定任务（如代码生成、对话）引入专用数据进行训练。\n",
"\n",
"---\n",
"\n",
"### **6. 实现持续预训练的代码示例**\n",
"\n",
"以下示例基于 Hugging Face 实现 GPT-2 的持续预训练：\n",
"\n",
"```python\n",
"from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments\n",
"from datasets import load_dataset\n",
"\n",
"# 1. 加载预训练模型和分词器\n",
"model_name = \"gpt2\"\n",
"tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
"model = AutoModelForCausalLM.from_pretrained(model_name)\n",
"\n",
"# 2. 加载新语料数据\n",
"dataset = load_dataset(\"text\", data_files={\"train\": \"domain_corpus.txt\"})\n",
"\n",
"# 3. 数据预处理\n",
"def tokenize_function(examples):\n",
" return tokenizer(examples[\"text\"], truncation=True, max_length=1024, padding=\"max_length\")\n",
"\n",
"tokenized_dataset = dataset.map(tokenize_function, batched=True)\n",
"\n",
"# 4. 设置训练参数\n",
"training_args = TrainingArguments(\n",
" output_dir=\"./gpt2_domain_adapted\",\n",
" overwrite_output_dir=True,\n",
" per_device_train_batch_size=4,\n",
" num_train_epochs=3,\n",
" learning_rate=5e-5,\n",
" save_steps=500,\n",
" save_total_limit=2,\n",
" logging_dir=\"./logs\",\n",
" evaluation_strategy=\"no\", # 评估策略可以根据需要调整\n",
" fp16=True, # 混合精度训练\n",
")\n",
"\n",
"# 5. 定义 Trainer 并启动训练\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_dataset[\"train\"],\n",
" tokenizer=tokenizer,\n",
")\n",
"\n",
"trainer.train()\n",
"\n",
"# 6. 保存模型\n",
"model.save_pretrained(\"./gpt2_domain_adapted\")\n",
"tokenizer.save_pretrained(\"./gpt2_domain_adapted\")\n",
"```\n",
"\n",
"---\n",
"\n",
"### **7. 持续预训练的挑战**\n",
"\n",
"1. **灾难性遗忘**：\n",
" - 持续预训练可能导致模型丧失之前学到的知识。\n",
" - **解决方法**：使用少量原始数据进行联合训练。\n",
"\n",
"2. **计算资源需求**：\n",
" - 需要大量显存和算力，特别是对于大规模模型和数据。\n",
"\n",
"3. **数据质量和多样性**：\n",
" - 新引入的数据可能包含噪声，影响模型性能。\n",
"\n",
"---\n",
"\n",
"### **8. 持续预训练的优势**\n",
"\n",
"- 提高特定领域或任务的性能。\n",
"- 更高效地利用已有模型权重，避免从头训练。\n",
"- 保留原始模型的通用能力，同时增强领域适应性。\n",
"\n",
"---\n",
"\n",
"### **总结**\n",
"\n",
"持续预训练是适配领域任务和提升模型性能的重要方法，通过引入新数据或优化模型训练策略，可以让大模型在特定场景中表现更优。配合参数高效微调方法（如 LoRA），还可显著降低计算开销，提升训练效率。这种技术在学术研究、工业应用和前沿领域（如法律、医学等）中均具有广泛价值。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ca41ad33-18fb-44da-8f79-0380b5c9dcaa",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "3038550c-cc92-45c9-8bb4-46c58688bfc5",
"metadata": {},
"source": [
"## 本节任务\n",
"本节任务是基于llama。训练一个能够处理dna和protein蛋白质数据的基础预训练大模型，数据为第一章中的预训练数据，包括英文数据。"
]
},
{
"cell_type": "markdown",
"id": "aec90d65-ac62-4394-a526-ca62d8bdbad4",
"metadata": {},
"source": [
"## 环境设置\n",
"并行环境对transformer、peft等的版本要求比较高，如果版本不匹配可能会出现各种异常问题\n",
"之前的课程，都是单GPU运行，一般不存在版本问题，默认安装的都是最新版本。但运行并行环境时，需要确认下版本再运行，本课程运行并行环境如下：\n",
"\n",
"* Python 3.12.3\n",
"* transformers 4.45.2\n",
"* peft 0.3.0.dev0\n",
"* deepspeed 0.15.2\n",
"* accelerate 1.0.0\n",
"\n",
"如果不是，可以重新安装即可：\n",
"```\n",
"pip install transformers==4.45.2 deepspeed==0.15.2 accelerate==1.0.0\n",
"\n",
"#peft参考使用的是chinese llama的版本，需要git安装\n",
"\n",
"git clone https://github.com/huggingface/peft.git\n",
"\n",
"cd peft\n",
"\n",
"git checkout 13e53fc\n",
"\n",
"pip install . \n",
"```\n",
"如果有环境问题，可以查看本目录下的pip_list.txt"
]
},
{
"cell_type": "markdown",
"id": "b1bd33b8-2e05-4b59-9d8f-c48de194cfd6",
"metadata": {},
"source": [
"## 代码运行\n",
"\n",
"```\n",
"# 复制第一章训练数据,包括dna，protein，还有英文数据，添加英文数据是为了避免遗忘问题\n",
"\n",
"mkdir train_data\n",
"cp ../01-data_env/data/*.txt train_data/\n",
"使用这些数据，6卡4090大概大致需要训练16个小时，autodl也需要近200块钱了。\n",
"\n",
"建议学习时，可以使用1/10的数据训练：\n",
"awk ‘NR%10==1’ dna_1g.txt > dna.txt\n",
"rm dna_1g.txt\n",
"其他2类数据依次类推\n",
"\n",
"这样大概需要2到3个小时就能训练完成了\n",
"\n",
"\n",
"#持续预训练\n",
"./run_pt.sh\n",
"\n",
"#合并模型\n",
"./merge_pt_model.sh\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "4960a36c-7529-4db8-b91d-df91245f79d9",
"metadata": {},
"source": [
"## 模型验证"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "69b3e97f-a801-4264-a651-a854bcfba9c6",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoConfig,AutoModel\n",
"from transformers import DataCollatorForLanguageModeling\n",
"from transformers import Trainer, TrainingArguments\n",
"from transformers import AutoConfig, AutoModelForCausalLM,LlamaForCausalLM,LlamaTokenizer\n",
"from tokenizers import Tokenizer\n",
"from datasets import load_dataset"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "339435d9-9379-4b30-ae8b-50feee1ba714",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"LlamaTokenizer(name_or_path='dnahlm-merge-hf', vocab_size=91643, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'pad_token': '</s>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={\n",
"\t0: AddedToken(\"<unk>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
"\t1: AddedToken(\"<s>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
"\t2: AddedToken(\"</s>\", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n",
"}"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer = LlamaTokenizer.from_pretrained(\"dnahlm-merge-hf\")\n",
"tokenizer.pad_token = tokenizer.eos_token\n",
"tokenizer"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "d0f154bb-b1ab-4611-a14c-9b403043fd96",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "342e4ab139b64bb78f0429c2f92c8310",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"LlamaForCausalLM(\n",
" (model): LlamaModel(\n",
" (embed_tokens): Embedding(91643, 4096, padding_idx=0)\n",
" (layers): ModuleList(\n",
" (0-31): 32 x LlamaDecoderLayer(\n",
" (self_attn): LlamaSdpaAttention(\n",
" (q_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (k_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (v_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (o_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (rotary_emb): LlamaRotaryEmbedding()\n",
" )\n",
" (mlp): LlamaMLP(\n",
" (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)\n",
" (up_proj): Linear(in_features=4096, out_features=11008, bias=False)\n",
" (down_proj): Linear(in_features=11008, out_features=4096, bias=False)\n",
" (act_fn): SiLU()\n",
" )\n",
" (input_layernorm): LlamaRMSNorm((4096,), eps=1e-06)\n",
" (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-06)\n",
" )\n",
" )\n",
" (norm): LlamaRMSNorm((4096,), eps=1e-06)\n",
" (rotary_emb): LlamaRotaryEmbedding()\n",
" )\n",
" (lm_head): Linear(in_features=4096, out_features=91643, bias=False)\n",
")"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = LlamaForCausalLM.from_pretrained(\"dnahlm-merge-hf\") #continue pretrain\n",
"model"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "792a9f78-1828-4695-9f6e-479a704ea7e8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"LlamaConfig {\n",
" \"_name_or_path\": \"dnahlm-merge-hf\",\n",
" \"architectures\": [\n",
" \"LlamaForCausalLM\"\n",
" ],\n",
" \"attention_bias\": false,\n",
" \"attention_dropout\": 0.0,\n",
" \"bos_token_id\": 1,\n",
" \"eos_token_id\": 2,\n",
" \"head_dim\": 128,\n",
" \"hidden_act\": \"silu\",\n",
" \"hidden_size\": 4096,\n",
" \"initializer_range\": 0.02,\n",
" \"intermediate_size\": 11008,\n",
" \"max_position_embeddings\": 2048,\n",
" \"mlp_bias\": false,\n",
" \"model_type\": \"llama\",\n",
" \"num_attention_heads\": 32,\n",
" \"num_hidden_layers\": 32,\n",
" \"num_key_value_heads\": 32,\n",
" \"pad_token_id\": 0,\n",
" \"pretraining_tp\": 1,\n",
" \"rms_norm_eps\": 1e-06,\n",
" \"rope_scaling\": null,\n",
" \"rope_theta\": 10000.0,\n",
" \"tie_word_embeddings\": false,\n",
" \"torch_dtype\": \"float16\",\n",
" \"transformers_version\": \"4.45.2\",\n",
" \"use_cache\": true,\n",
" \"vocab_size\": 91643\n",
"}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import AutoConfig\n",
"# 加载配置\n",
"config = AutoConfig.from_pretrained('dnahlm-merge-hf')\n",
"config"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "49021c65-54bb-4a97-a96d-b030cc3dcd13",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test text:\n",
" GCTGACTCTGCCAGGATGGAATGAAATTAGGTTGTTTTAATTATAATGTAAAGTCAGTTCTAGTCAGACATAGTCACATAGGCAAGTAAGGGAACCTAAAATTGCTTGGAAT,\n",
"KCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKEGIAITGCWPRWPDEMDERSAVWRVEPYTRHFGRVLYSFGV,\n",
"The primary use of LLaMA is research on large language models, including\n",
"Tokenized by DNA-LLaMA tokenizer:['▁GC', 'TGA', 'CT', 'C', 'TGCC', 'AGGATGG', 'AATG', 'AAATT', 'AGGTTG', 'TTTTAATT', 'ATAATGTAA', 'AGTCAG', 'TTCTAG', 'TCAG', 'ACATAG', 'TC', 'ACATAGG', 'CA', 'AGTAAGGG', 'AAC', 'CT', 'AAAATTGC', 'TTGG', 'AAT', ',', '<0x0A>', 'KCG', 'FVGP', 'MVHL', 'KV', 'HLE', 'ADV', 'ASSC', 'RSAV', 'I', 'YL', 'TSEE', 'P', 'FEG', 'VLGL', 'RLK', 'EGI', 'AI', 'TGC', 'W', 'PRW', 'P', 'DEM', 'DER', 'SAV', 'W', 'RVE', 'PY', 'TRH', 'FG', 'RVLY', 'SFGV', ',', '<0x0A>', 'The', '▁primary', '▁use', '▁of', '▁L', 'La', 'MA', '▁is', '▁research', '▁on', '▁large', '▁language', '▁models', ',', '▁including']\n"
]
}
],
"source": [
"text='''GCTGACTCTGCCAGGATGGAATGAAATTAGGTTGTTTTAATTATAATGTAAAGTCAGTTCTAGTCAGACATAGTCACATAGGCAAGTAAGGGAACCTAAAATTGCTTGGAAT,\n",
"KCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKEGIAITGCWPRWPDEMDERSAVWRVEPYTRHFGRVLYSFGV,\n",
"The primary use of LLaMA is research on large language models, including'''\n",
"print(\"Test text:\\n\",text)\n",
"print(f\"Tokenized by DNA-LLaMA tokenizer:{tokenizer.tokenize(text)}\")"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "ebf869c8-866d-4770-8f64-79d671f88663",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e497889a1c3c484cb57c4b6fd93b45ab",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some parameters are on the meta device because they were offloaded to the cpu.\n",
"/root/miniconda3/lib/python3.12/site-packages/transformers/generation/utils.py:1220: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
" warnings.warn(\n",
"Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)\n"
]
},
{
"data": {
"text/plain": [
"[{'generated_text': 'The key to life is to accept the fact that you are going to die. The key to'}]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import torch\n",
"from transformers import pipeline\n",
"\n",
"model_id = \"dnahlm-merge-hf\"\n",
"\n",
"pipe = pipeline(\n",
" \"text-generation\", \n",
" model=model_id, \n",
" #torch_dtype=torch.bfloat16, \n",
" device_map=\"auto\",\n",
")\n",
"\n",
"pipe(\"The key to life is\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "40a22c70-f1c4-4cd5-a118-2f5db40790e6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'generated_text': 'GGAATGAAATTAGGTTGTTTTAATTATAATGTAAAGTCAGTTCTCTCCTCCTCCTCCTC'}]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pipe(\"GGAATGAAATTAGGTTGTTTTAATTATAATGTAAAGTCAGTTCT\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "aec95d0a-4269-4540-bf14-4ce157b9a194",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'generated_text': 'KCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKETLK'}]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pipe(\"KCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLK\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c1cfab60-2820-4885-8961-0290c49dfbec",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/7-llama-instruction-ft-checkpoint.ipynb
+998
View File
@@ -0,0 +1,998 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "963e9ae0-ac68-44be-8c7d-fb9842784362",
"metadata": {},
"source": [
"# 4.7 基于llama的基因大模型指令微调"
]
},
{
"cell_type": "markdown",
"id": "c844103d-4e27-41b9-9bf1-c6a577846ab6",
"metadata": {},
"source": [
"### **大模型的指令微调（Instruction Fine-Tuning）**\n",
"\n",
"指令微调是指通过对大语言模型（如 GPT、T5、LLaMA 等）进行微调，使其能够更好地理解和执行人类以指令形式表达的任务。这种技术是大模型适配实际应用和增强用户交互能力的关键手段。\n",
"\n",
"---\n",
"\n",
"### **1. 指令微调的核心概念**\n",
"\n",
"指令微调的目标是通过在包含指令的专用数据集上进行微调，让模型能够：\n",
"1. 理解用户的任务需求（以自然语言表达的指令形式）。\n",
"2. 根据指令内容生成符合预期的高质量响应。\n",
"3. 适应多任务场景，减少特定任务的单独训练需求。\n",
"\n",
"---\n",
"\n",
"### **2. 指令微调的关键特点**\n",
"\n",
"1. **多任务统一**：\n",
" - 不需要针对每个任务单独微调，而是通过指令微调使模型能适应多种任务。\n",
" \n",
"2. **自然语言交互**：\n",
" - 用户可以用自然语言指令与模型交互，无需提供特定格式的输入。\n",
"\n",
"3. **泛化能力**：\n",
" - 微调后的模型能够对未见过的任务产生合理的推断和响应。\n",
"\n",
"---\n",
"\n",
"### **3. 数据集的构建与使用**\n",
"\n",
"#### **（1）指令微调数据集的特点**\n",
"- 数据通常包含以下三部分：\n",
" 1. **指令（Instruction）**：任务描述或问题，例如“将以下文本翻译为法语”。\n",
" 2. **输入（Input）**：任务相关的上下文或数据，可以为空。\n",
" 3. **输出（Output）**：模型期望生成的结果。\n",
"\n",
"#### **（2）常用指令微调数据集**\n",
"- **FLAN**：包含多个 NLP 任务的指令数据集，用于 T5 等模型的微调。\n",
"- **OpenAI 提供的指令数据**：如 GPT 系列的 ChatGPT 调优数据集。\n",
"- **InstructGPT 数据**：通过人类标注的多任务指令数据，用于模型优化。\n",
"- **Self-Instruct**：通过模型自生成指令和回答，进一步扩展训练数据。\n",
"\n",
"#### **（3）构建自己的数据集**\n",
"- 如果需要特定领域的指令微调，可以自行构建数据集：\n",
" - 收集任务需求和示例。\n",
" - 设计多样化的指令。\n",
" - 使用专家标注或模型辅助生成高质量答案。\n",
"\n",
"---\n",
"\n",
"### **4. 微调的步骤**\n",
"\n",
"#### **（1）加载基础模型**\n",
"从 Hugging Face 或其他框架加载预训练的大语言模型，例如 GPT-2、T5、LLaMA。\n",
"\n",
"#### **（2）准备数据集**\n",
"将指令微调数据集格式化为：\n",
"```python\n",
"{\n",
" \"instruction\": \"Translate the following text to French\",\n",
" \"input\": \"Hello, how are you?\",\n",
" \"output\": \"Bonjour, comment ça va?\"\n",
"}\n",
"```\n",
"\n",
"#### **（3）定义微调方法**\n",
"使用 `Trainer` 或分布式框架（如 DeepSpeed、Accelerate）进行微调。\n",
"\n",
"---\n",
"\n",
"### **5. 示例代码：指令微调实现**\n",
"\n",
"以下是基于 Hugging Face 的指令微调代码示例：\n",
"\n",
"```python\n",
"from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer\n",
"from datasets import load_dataset\n",
"\n",
"# 1. 加载预训练模型和分词器\n",
"model_name = \"gpt2\"\n",
"tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
"model = AutoModelForCausalLM.from_pretrained(model_name)\n",
"\n",
"# 2. 加载指令微调数据集\n",
"# 数据格式应包含 instruction, input, output 字段\n",
"dataset = load_dataset(\"path/to/instruction_dataset\")\n",
"\n",
"# 3. 数据预处理\n",
"def preprocess_function(example):\n",
" # 将指令和输入拼接成完整的提示\n",
" prompt = example[\"instruction\"]\n",
" if example[\"input\"]:\n",
" prompt += f\"\\n{example['input']}\"\n",
" labels = example[\"output\"]\n",
" tokenized = tokenizer(prompt, truncation=True, max_length=512, padding=\"max_length\")\n",
" with tokenizer.as_target_tokenizer():\n",
" tokenized_labels = tokenizer(labels, truncation=True, max_length=512, padding=\"max_length\")\n",
" tokenized[\"labels\"] = tokenized_labels[\"input_ids\"]\n",
" return tokenized\n",
"\n",
"tokenized_datasets = dataset.map(preprocess_function, batched=True)\n",
"\n",
"# 4. 设置训练参数\n",
"training_args = TrainingArguments(\n",
" output_dir=\"./instruction_finetuned_model\",\n",
" per_device_train_batch_size=4,\n",
" num_train_epochs=3,\n",
" evaluation_strategy=\"epoch\",\n",
" save_strategy=\"epoch\",\n",
" learning_rate=5e-5,\n",
" weight_decay=0.01,\n",
" logging_dir=\"./logs\",\n",
" fp16=True,\n",
")\n",
"\n",
"# 5. 定义 Trainer\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"test\"],\n",
" tokenizer=tokenizer,\n",
")\n",
"\n",
"# 6. 开始训练\n",
"trainer.train()\n",
"\n",
"# 7. 保存模型\n",
"model.save_pretrained(\"./instruction_finetuned_model\")\n",
"tokenizer.save_pretrained(\"./instruction_finetuned_model\")\n",
"```\n",
"\n",
"---\n",
"\n",
"### **6. 指令微调的挑战**\n",
"\n",
"1. **数据质量**：\n",
" - 低质量或噪声数据可能导致模型生成结果不符合指令。\n",
"\n",
"2. **指令覆盖范围**：\n",
" - 数据集指令种类不足会限制模型的泛化能力。\n",
"\n",
"3. **计算资源需求**：\n",
" - 大模型的微调需要高性能 GPU 和大容量存储。\n",
"\n",
"4. **灾难性遗忘**：\n",
" - 微调过程中可能导致模型丧失部分原始能力。\n",
"\n",
"---\n",
"\n",
"### **7. 指令微调的应用场景**\n",
"\n",
"1. **多任务问答**：\n",
" - 适配多任务场景，支持翻译、总结、推理等功能。\n",
"\n",
"2. **特定领域优化**：\n",
" - 在法律、医疗等特定领域的任务指令上进行微调。\n",
"\n",
"3. **用户交互优化**：\n",
" - 提升模型对自然语言指令的理解和响应能力。\n",
"\n",
"4. **开放式对话生成**：\n",
" - 优化模型在对话场景下的表现，例如 ChatGPT 的微调。\n",
"\n",
"---\n",
"\n",
"### **总结**\n",
"\n",
"指令微调通过在特定格式的数据集上进一步训练大模型，使其能够更好地理解和执行用户的自然语言指令。这种方法适合多任务场景，并能提升模型的交互能力和领域适应性。借助高质量的指令数据集和高效的微调技术，大模型在实际应用中的表现可以得到显著提升。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e77f8b39-e75a-4014-a98a-bde5b2534bf1",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "7be8b814-42f6-4fb6-bf4b-ae23292030f6",
"metadata": {},
"source": [
"## 持续预训练 VS 指令微调"
]
},
{
"cell_type": "markdown",
"id": "f9bed0ae-337d-49af-85f0-c8e6263d78db",
"metadata": {},
"source": [
"**大模型的持续预训练**和**指令微调**是两种针对大模型的后续优化策略，虽然它们的目标都是提升模型性能，但在应用场景、方法和效果等方面有明显区别。以下是它们的对比分析：\n",
"\n",
"---\n",
"\n",
"### **1. 概念与目标**\n",
"\n",
"| **特性** | **持续预训练** | **指令微调** |\n",
"|------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|\n",
"| **定义** | 在通用预训练模型上，使用新的大规模语料（通用或领域特定数据）进行进一步预训练。 | 在包含指令任务的数据集上对大模型进行微调，以提升模型对人类指令的理解和执行能力。 |\n",
"| **目标** | 提升模型的通用能力或适应特定领域的语言理解与生成能力。 | 提高模型对多任务指令的泛化能力，让模型更好地理解和执行自然语言表达的具体任务。 |\n",
"| **典型应用** | 领域适配（医学、法律、金融）、性能优化、跨语言适配等。 | 多任务问答、开放式对话生成、翻译、推理等需要用户直接交互的场景。 |\n",
"\n",
"---\n",
"\n",
"### **2. 数据使用**\n",
"\n",
"| **特性** | **持续预训练** | **指令微调** |\n",
"|------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|\n",
"| **数据类型** | 通用语料（如新闻、社交媒体文本）或领域特定语料（如 PubMed、法律文档、金融报告）。 | 任务指令数据集，包括指令（Instruction）、输入（Input）和输出（Output）。 |\n",
"| **数据构建** | 通常需要清洗和去重大规模语料数据，避免与原始预训练数据重叠。 | 通常由人工标注或模型生成的指令数据构成，例如 FLAN、InstructGPT 数据集。 |\n",
"| **多样性要求** | 数据应覆盖尽可能广的领域或目标领域的多种场景，以提升模型在这些场景的表现。 | 数据需要覆盖多种任务类型（如翻译、分类、摘要）和丰富的指令表达形式，以提高模型对多任务的适配能力。 |\n",
"\n",
"---\n",
"\n",
"### **3. 方法与技术**\n",
"\n",
"| **特性** | **持续预训练** | **指令微调** |\n",
"|------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|\n",
"| **主要技术** | 继续使用自监督学习目标（如语言建模、掩码预测）进行训练。 | 使用监督学习，通常以任务输入和目标输出对为数据，通过微调适配特定任务需求。 |\n",
"| **模型调整** | - 可选择全量参数更新或冻结部分参数。<br>- 可结合参数高效微调技术（如 LoRA、Adapter）。 | - 通常使用监督训练方式，可能结合参数高效微调技术（如 LoRA）。 |\n",
"| **学习率** | 通常使用较小的学习率（如 `1e-5` 或更小），以防止破坏原始权重。 | 同样使用较小的学习率，但任务指令微调可能需要更高的关注任务特定的标签对准。 |\n",
"\n",
"---\n",
"\n",
"### **4. 模型能力与效果**\n",
"\n",
"| **特性** | **持续预训练** | **指令微调** |\n",
"|------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|\n",
"| **提升的能力** | - 对领域特定语言模式和知识的适配性提升显著。<br>- 对未见过的通用场景生成能力增强（扩展模型知识广度）。 | - 显著提升模型对指令理解的能力，尤其是自然语言表达的任务需求。<br>- 对多任务和零样本任务的泛化能力有较大提升。 |\n",
"| **局限性** | - 对具体任务的直接适配能力较弱，可能需要额外的任务微调。<br>- 数据选择不当可能导致灾难性遗忘。 | - 依赖高质量的指令数据集，数据质量不高会导致模型生成结果不稳定。<br>- 对通用能力的提升有限。 |\n",
"\n",
"---\n",
"\n",
"### **5. 应用场景与示例**\n",
"\n",
"| **特性** | **持续预训练** | **指令微调** |\n",
"|------------------------|-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|\n",
"| **典型应用场景** | - 医学文献总结（通过 PubMed 语料持续预训练）。<br>- 法律条文分析（通过法律文档进一步训练）。<br>- 增强多语言生成能力（跨语言语料）。 | - ChatGPT 的多任务对话生成。<br>- 翻译、摘要、问答等用户交互任务的泛化处理。 |\n",
"| **实际示例** | - BioBERT：在 BERT 基础上使用生物医学语料持续预训练的模型。<br>- FinBERT：针对金融领域持续预训练的语言模型。 | - InstructGPT：在 GPT-3 基础上进行指令微调，用于多任务用户交互。<br>- FLAN-T5：通过 FLAN 数据集进行指令微调。 |\n",
"\n",
"---\n",
"\n",
"### **6. 持续预训练与指令微调的结合**\n",
"\n",
"持续预训练和指令微调可以结合使用，形成一个从领域适配到任务适配的完整流程：\n",
"1. **持续预训练**：\n",
" - 先在领域特定数据（如医学、法律、金融语料）上进行持续预训练，获取领域知识。\n",
"2. **指令微调**：\n",
" - 再利用多任务指令数据集对模型微调，使其能够高效执行领域内的多样化任务。\n",
"\n",
"这种结合方式特别适用于需要领域知识和任务适配的场景，例如医学问答系统或金融文本分析。\n",
"\n",
"---\n",
"\n",
"### **总结**\n",
"\n",
"| **维度** | **持续预训练** | **指令微调** |\n",
"|------------------------|-------------------------------------|----------------------------------|\n",
"| **目标** | 增强通用能力或适配特定领域。 | 提升对任务指令的理解和执行能力。 |\n",
"| **数据集** | 通用或领域语料。 | 指令数据集，包含输入和输出对。 |\n",
"| **方法** | 自监督学习，扩展语言建模能力。 | 监督学习，强化任务适配能力。 |\n",
"| **适用场景** | 领域特定任务（如医学、法律）。 | 多任务交互（如问答、对话生成）。 |\n",
"| **局限性** | 对具体任务适配较弱。 | 通用能力提升有限，依赖数据质量。 |\n",
"\n",
"两者各有侧重，且在许多场景下可以结合使用，形成一个强大的任务和领域适配框架。"
]
},
{
"cell_type": "markdown",
"id": "f97a705a-b946-4dc1-a173-a9df033d6f2b",
"metadata": {},
"source": [
"## 本节任务\n",
"本节任务是基于上一节预训练的llama生物大模型。对一些生物学任务进行微调，包含了多个不同类型的分类问题和多序列交换问题。具体可见sft_data下的数据。"
]
},
{
"cell_type": "markdown",
"id": "9782db62-95bd-40a6-9759-966b9a0b362e",
"metadata": {},
"source": [
"## 代码运行\n",
"\n",
"```\n",
"\n",
"#微调\n",
"./run_sft.sh\n",
"\n",
"运行时间约3小时\n",
"\n",
"#合并模型\n",
"./merge_sft_model.sh\n",
"\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "182b82c4-d484-4c15-a600-03c3b51367ec",
"metadata": {},
"source": [
"## 模型验证"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5aa3d240-44e1-4811-8f61-d6ff2500a798",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "054a2956-9045-4ad5-a878-1bfc84ad4ed8",
"metadata": {},
"outputs": [],
"source": [
"from transformers import AutoTokenizer, AutoConfig,AutoModel\n",
"from transformers import DataCollatorForLanguageModeling\n",
"from transformers import Trainer, TrainingArguments\n",
"from transformers import AutoConfig, AutoModelForCausalLM,LlamaForCausalLM,LlamaTokenizer\n",
"from tokenizers import Tokenizer\n",
"from datasets import load_dataset"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "63c8bf16-9576-41bc-b27c-c92ba4289cf4",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['instruction', 'input', 'output'],\n",
" num_rows: 19839\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from datasets import load_dataset\n",
"dna_ft_dataset = load_dataset('json', data_files='val_data.json')\n",
"dna_ft_dataset"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "95928da3-ca64-4a17-80f4-945da395702c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['instruction', 'input', 'output'],\n",
" num_rows: 1983\n",
" })\n",
" test: Dataset({\n",
" features: ['instruction', 'input', 'output'],\n",
" num_rows: 17856\n",
" })\n",
"})"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = dna_ft_dataset[\"train\"].train_test_split(train_size=0.1, seed=42)\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a3e65bcd-85ce-4261-8ba6-7665c4ec60e2",
"metadata": {},
"outputs": [],
"source": [
"tokenizer = LlamaTokenizer.from_pretrained(\"dnahlm-llama-7b-sft-v0\") #dnagpt/dnahlm-llama-7b-sft-v0\n",
"tokenizer.pad_token = tokenizer.eos_token"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "3d3fe49b-f48f-42b2-bc97-028e443111e4",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4f060ff2029447b9bad5e2b2e40b7133",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"LlamaForCausalLM(\n",
" (model): LlamaModel(\n",
" (embed_tokens): Embedding(91644, 4096, padding_idx=0)\n",
" (layers): ModuleList(\n",
" (0-31): 32 x LlamaDecoderLayer(\n",
" (self_attn): LlamaSdpaAttention(\n",
" (q_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (k_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (v_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (o_proj): Linear(in_features=4096, out_features=4096, bias=False)\n",
" (rotary_emb): LlamaRotaryEmbedding()\n",
" )\n",
" (mlp): LlamaMLP(\n",
" (gate_proj): Linear(in_features=4096, out_features=11008, bias=False)\n",
" (up_proj): Linear(in_features=4096, out_features=11008, bias=False)\n",
" (down_proj): Linear(in_features=11008, out_features=4096, bias=False)\n",
" (act_fn): SiLU()\n",
" )\n",
" (input_layernorm): LlamaRMSNorm((4096,), eps=1e-06)\n",
" (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-06)\n",
" )\n",
" )\n",
" (norm): LlamaRMSNorm((4096,), eps=1e-06)\n",
" (rotary_emb): LlamaRotaryEmbedding()\n",
" )\n",
" (lm_head): Linear(in_features=4096, out_features=91644, bias=False)\n",
")"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model = LlamaForCausalLM.from_pretrained(\"dnahlm-llama-7b-sft-v0\") #continue pretrain\n",
"model"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "c54df9fe-86c4-4963-b313-b438894bf9dd",
"metadata": {},
"outputs": [],
"source": [
"#构建提示词\n",
"def format_input(entry):\n",
" instruction_text = (\n",
" f\"Below is an instruction that describes a task. \"\n",
" f\"Write a response that appropriately completes the request.\"\n",
" f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
" )\n",
"\n",
" input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
"\n",
" return instruction_text + input_text + \"\\n\\n### Response:\\n\"\n",
"\n",
"#构建提示词\n",
"def build_prompt(entry):\n",
"\n",
" input_data = format_input(entry)\n",
"\n",
" desired_response = entry['output']\n",
"\n",
" return input_data + desired_response\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "ee540cfb-1f6e-4e02-a3bc-c814e43685cb",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'instruction': 'Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.',\n",
" 'input': 'CCGTGCGACCGGAAGTGGGGCGGCGACCCCGGAAGTCCCCGCCGGGTGCAGCTTGGTCGGTTCGATCGCC',\n",
" 'output': 'promoter'}"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example = data[\"test\"][0]\n",
"example"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7ee35528-7b3f-4e60-b88b-1bc3e950012b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
"\n",
"### Instruction:\n",
"Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.\n",
"\n",
"### Input:\n",
"CCGTGCGACCGGAAGTGGGGCGGCGACCCCGGAAGTCCCCGCCGGGTGCAGCTTGGTCGGTTCGATCGCC\n",
"\n",
"### Response:\n",
"promoter\n"
]
}
],
"source": [
"prompt = build_prompt(example)\n",
"print(prompt)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "8aa6f38f-3bcc-4566-8a66-a541db91e031",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['▁Below',\n",
" '▁is',\n",
" '▁an',\n",
" '▁instruction',\n",
" '▁that',\n",
" '▁describes',\n",
" '▁a',\n",
" '▁task',\n",
" '.',\n",
" '▁Write',\n",
" '▁a',\n",
" '▁response',\n",
" '▁that',\n",
" '▁appropri',\n",
" 'ately',\n",
" '▁comple',\n",
" 'tes',\n",
" '▁the',\n",
" '▁request',\n",
" '.',\n",
" '<0x0A>',\n",
" '<0x0A>',\n",
" '##',\n",
" '#',\n",
" '▁Inst',\n",
" 'ruction',\n",
" ':',\n",
" '<0x0A>',\n",
" 'Det',\n",
" 'erm',\n",
" 'ine',\n",
" '▁core',\n",
" '▁prom',\n",
" 'oter',\n",
" '▁detection',\n",
" '▁of',\n",
" '▁following',\n",
" '▁d',\n",
" 'na',\n",
" '▁sequence',\n",
" ',',\n",
" '▁The',\n",
" '▁result',\n",
" '▁will',\n",
" '▁be',\n",
" '▁one',\n",
" '▁of',\n",
" '▁the',\n",
" '▁following',\n",
" ':',\n",
" '▁Non',\n",
" '-',\n",
" 'prom',\n",
" 'oter',\n",
" ',',\n",
" '▁prom',\n",
" 'oter',\n",
" '.',\n",
" '<0x0A>',\n",
" '<0x0A>',\n",
" '##',\n",
" '#',\n",
" '▁Input',\n",
" ':',\n",
" '<0x0A>',\n",
" 'CCG',\n",
" 'TGCG',\n",
" 'ACCGG',\n",
" 'AAG',\n",
" 'TGGGGC',\n",
" 'GGCG',\n",
" 'ACCCCGG',\n",
" 'AAG',\n",
" 'TCCCC',\n",
" 'GCCGGG',\n",
" 'TGCAGC',\n",
" 'TTGG',\n",
" 'TCGG',\n",
" 'TTCG',\n",
" 'ATCGCC',\n",
" '<0x0A>',\n",
" '<0x0A>',\n",
" '##',\n",
" '#',\n",
" '▁Response',\n",
" ':',\n",
" '<0x0A>',\n",
" 'prom',\n",
" 'oter']"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tokenizer.tokenize(prompt)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "11875339-4901-4912-86e5-afe8c74921d9",
"metadata": {},
"outputs": [],
"source": [
"def inference(text, model, tokenizer, max_input_tokens=1000, max_output_tokens=1000):\n",
" # Tokenize\n",
" input_ids = tokenizer.encode(\n",
" text,\n",
" return_tensors=\"pt\",\n",
" truncation=True,\n",
" max_length=max_input_tokens\n",
" # return_attention_mask=True,\n",
" )\n",
"\n",
" # Generate\n",
" device = model.device\n",
" generated_tokens_with_prompt = model.generate(\n",
" input_ids=input_ids.to(device),\n",
" #max_length=max_output_tokens,\n",
" max_new_tokens=8,\n",
" temperature=0.01 # 控制生成的多样性\n",
" )\n",
"\n",
" # Decode\n",
" generated_text_with_prompt = tokenizer.decode(generated_tokens_with_prompt[0], skip_special_tokens=True)\n",
" generated_text_answer = generated_text_with_prompt[len(text):]\n",
"\n",
"\n",
" return generated_text_answer\n",
"\n",
"# 如果需要进一步清理\n",
"def clean_generated_text(text):\n",
" # 去除 'Ġ' 符号并替换为空格\n",
" text = text.replace('Ġ', ' ')\n",
" # 去除多余的空格\n",
" text = ' '.join(text.split())\n",
" return text"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "1b02644a-8b24-45aa-b22d-0f7ce2270dd9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"input (test): Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
"\n",
"### Instruction:\n",
"Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.\n",
"\n",
"### Input:\n",
"CCGTGCGACCGGAAGTGGGGCGGCGACCCCGGAAGTCCCCGCCGGGTGCAGCTTGGTCGGTTCGATCGCC\n",
"\n",
"### Response:\n",
"\n",
"real answer: promoter\n",
"--------------------------\n",
"\n",
"model's answer: \n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/root/miniconda3/lib/python3.12/site-packages/transformers/generation/configuration_utils.py:601: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.01` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
" warnings.warn(\n",
"Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"promoter\n"
]
}
],
"source": [
"input_text = format_input(data[\"test\"][0])\n",
"\n",
"print(\"input (test):\", input_text)\n",
"\n",
"print(\"real answer:\", data[\"test\"][0][\"output\"])\n",
"\n",
"print(\"--------------------------\\n\")\n",
"\n",
"print(\"model's answer: \\n\")\n",
"print(inference(input_text, model, tokenizer))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e2df1569-7f70-46ee-b93f-cbd879e32e14",
"metadata": {},
"outputs": [],
"source": [
"test_data = data[\"test\"].shuffle(seed=199).select(range(100))\n",
"\n",
"data_list = []\n",
"\n",
"for entry in test_data:\n",
" input_text = format_input(entry)\n",
" #print(input_text)\n",
" response_text = inference(input_text, model, tokenizer)\n",
" #print(response_text)\n",
" data = {\n",
" \"instruction\":entry[\"instruction\"],\n",
" \"input\":entry[\"input\"],\n",
" \"output\":entry[\"output\"],\n",
" \"model_response\":response_text\n",
" }\n",
"\n",
" data_list.append(data)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0c6e47cb-1b64-4690-a51d-f1816b82f15f",
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"\n",
"# 定义输出文件路径\n",
"output_file = 'llama-sft-2.json'\n",
"\n",
"# 将 Dataset 对象导出为 JSON 文件\n",
"# test_data.to_json(output_file)\n",
"with open(output_file, \"w\") as file:\n",
" json.dump(data_list, file, indent=4) # \"indent\" for pretty-printing\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "68831e19-5a99-46d8-9f40-e8bf6957dbfc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Donor Sites |||||||||||| Non-Splice Sites\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"Donor Sites |||||||||||| Non-Splice Sites\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Donor Sites |||||||||||| Splice Sites\n",
"Background Sequences |||||||||||| Background Sequences\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-Splice Sites |||||||||||| Non-Splice Sites\n",
"Non-promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Binding Sites |||||||||||| Background Sequences\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-Splice Sites |||||||||||| Non-Splice Sites\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Non-promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Background Sequences |||||||||||| Background Sequences\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Binding Sites |||||||||||| Binding Sites\n",
"promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Donor Sites |||||||||||| Donor Sites\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Binding Sites |||||||||||| Binding Sites\n",
"promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"Background Sequences |||||||||||| Binding Sites\n",
"promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Background Sequences |||||||||||| Background Sequences\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"Donor Sites |||||||||||| Non-Splice Sites\n",
"Binding Sites |||||||||||| Binding Sites\n",
"promoter |||||||||||| promoter\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Non-promoter |||||||||||| promoter\n",
"Binding Sites |||||||||||| Binding Sites\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Non-promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Acceptor Sites |||||||||||| Acceptor Sites\n",
"promoter |||||||||||| promoter\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Donor Sites |||||||||||| Acceptor Sites\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| promoter\n",
"promoter |||||||||||| Non-promoter\n",
"Non-promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"Background Sequences |||||||||||| Binding Sites\n",
"Acceptor Sites |||||||||||| Splice Sites\n",
"Non-Splice Sites |||||||||||| Non-Splice Sites\n",
"Donor Sites |||||||||||| Non-Splice Sites\n",
"Donor Sites |||||||||||| Donor Sites\n",
"Non-promoter |||||||||||| Non-promoter\n",
"promoter |||||||||||| promoter\n",
"Background Sequences |||||||||||| Binding Sites\n",
"promoter |||||||||||| promoter\n",
"promoter |||||||||||| promoter\n",
"Acceptor Sites |||||||||||| Splice Sites\n",
"promoter |||||||||||| promoter\n",
"presicion 0.73 same 0.3\n"
]
}
],
"source": [
"import json\n",
"from tqdm import tqdm\n",
"\n",
"\n",
"\n",
"with open(output_file, \"r\") as file:\n",
" test_data = json.load(file)\n",
"\n",
"all_num = len(test_data)\n",
"right_sum = 0\n",
"same_sum = 0\n",
"for item in test_data:\n",
" output = item[\"output\"]\n",
" #output = \" \".join(tokenizer.tokenize(output))\n",
" model_response = item[\"model_response\"]\n",
"\n",
" print(output,\"||||||||||||\", model_response)\n",
"\n",
" if model_response == output: #same it\n",
" same_sum = same_sum + 1\n",
" \n",
" if output.find(\"Non\")==-1: # no Non\n",
" if model_response.find(output)!=-1 and model_response.find(\"Non\")==-1: #find it, but no Non\n",
" right_sum = right_sum + 1\n",
" else:\n",
" if model_response.find(output)!=-1: #find it\n",
" right_sum = right_sum + 1\n",
"\n",
"\n",
"print(\"presicion\", right_sum/all_num, \"same\", same_sum/all_num)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7bc38f47-4a7d-43eb-abe8-db4310d280e3",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/build_gene_bpe_seg-checkpoint.py
+14
View File
@@ -0,0 +1,14 @@
import sentencepiece as spm
spm.SentencePieceTrainer.train(input='../01-data_env/data/dna_1g.txt,../01-data_env/data/protein_1g.txt',
model_prefix='gene_bpe_seg',
vocab_size=60000,
model_type='bpe', #默认是unigram
num_threads=10,
)
from sentencepiece import SentencePieceProcessor
model_path = "gene_bpe_seg.model"
sp_model = SentencePieceProcessor(model_file=model_path)
mm = sp_model.EncodeAsPieces("TCGACGGCACGCGACAGCAGCGAGCCCCGCGCACCCGAGCGCGAKCGFVGPMVHLKVHLEADVASSCRSAVIYLTSEEPFEGVLGLRLKEGIAITGCWPRWPDEMDERSAVWRVEPYTRHFGRVLYSFGV")
print(mm)
04-gene-sft/.ipynb_checkpoints/deepspeed_pretrain_gpt2-checkpoint.py
+114
View File
@@ -0,0 +1,114 @@
# import os
# # 设置环境变量
# os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
# # 打印环境变量以确认设置成功
# print(os.environ.get('HF_ENDPOINT'))
import subprocess
import os
# 设置环境变量, autodl一般区域
result = subprocess.run('bash -c "source /etc/network_turbo && env | grep proxy"', shell=True, capture_output=True, text=True)
output = result.stdout
for line in output.splitlines():
if '=' in line:
var, value = line.split('=', 1)
os.environ[var] = value
import math
from transformers import (
GPT2Config,
GPT2LMHeadModel,
GPT2TokenizerFast,
TrainingArguments,
Trainer,
DataCollatorForLanguageModeling,
)
from datasets import Dataset
from datasets import load_dataset
import evaluate
import numpy as np
from transformers import AutoTokenizer,AutoConfig
# 加载 OpenWebText 数据集
dataset = load_dataset("text", data_files="../01-data_env/data/dna_1g.txt")["train"].train_test_split(test_size=0.01, shuffle=True)
# 定义最大输入长度
max_length = 256
# 数据预处理
def preprocess_function(examples):
return tokenizer(examples["text"], truncation=True, max_length=max_length)
# 初始化 GPT-2 分词器
tokenizer = AutoTokenizer.from_pretrained("gpt2_tokenizer")
tokenized_dataset = dataset.map(preprocess_function, batched=True, num_proc=5)
# 4. 创建一个数据收集器，用于动态填充和遮蔽
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer, mlm=False
)
# 加载并调整 GPT-2 配置
config = AutoConfig.from_pretrained(
"gpt2", # 加载 GPT-2 的默认配置
vocab_size=len(tokenizer), # 更新词汇表大小为自定义分词器的词汇表大小
n_ctx=max_length, # 最大上下文长度（序列长度）
n_positions=max_length, # 最大位置编码长度，通常与 n_ctx 一致
)
# 初始化 GPT-2 模型
model = GPT2LMHeadModel(config)
# 定义训练参数
training_args = TrainingArguments(
output_dir="./gpt2-small",
overwrite_output_dir=True,
num_train_epochs=5,
per_device_train_batch_size=64,
save_steps=10000,
save_total_limit=2,
logging_dir="./logs",
logging_steps=20000,
evaluation_strategy="steps",
eval_steps=10000,
learning_rate=5e-5,
warmup_steps=500,
weight_decay=0.01,
fp16=True, # 启用混合精度训练
deepspeed="ds_zero2_no_offload.json"
)
# 初始化 Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset["train"],
eval_dataset=tokenized_dataset["test"],
tokenizer=tokenizer,
data_collator=data_collator,
)
# 开始训练
trainer.train()
# 评估 perplexity
eval_results = trainer.evaluate()
perplexity = math.exp(eval_results["eval_loss"])
print(f"Perplexity: {perplexity}")
out_model_path = "gpt2-small-gene-openweb"
trainer.save_model(out_model_path)
tokenizer.save_pretrained(out_model_path)
04-gene-sft/.ipynb_checkpoints/ds_zero2_no_offload-checkpoint.json
+27
View File
@@ -0,0 +1,27 @@
{
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"loss_scale_window": 100,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1e-10
},
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
"allgather_bucket_size": 1e8,
"overlap_comm": true,
"reduce_scatter": true,
"reduce_bucket_size": 1e8,
"contiguous_gradients": true
},
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"steps_per_print": 2000,
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false
}
04-gene-sft/.ipynb_checkpoints/env_ini-checkpoint.ipynb
+267
View File
@@ -0,0 +1,267 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 5,
"id": "e3fbdac5-cd38-4e41-b5d2-d9d112b4ac1b",
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: http://mirrors.aliyun.com/pypi/simple\n",
"Requirement already satisfied: transformers in /root/miniconda3/lib/python3.12/site-packages (4.47.1)\n",
"Requirement already satisfied: sentencepiece in /root/miniconda3/lib/python3.12/site-packages (0.2.0)\n",
"Requirement already satisfied: google in /root/miniconda3/lib/python3.12/site-packages (3.0.0)\n",
"Requirement already satisfied: protobuf in /root/miniconda3/lib/python3.12/site-packages (5.27.0)\n",
"Requirement already satisfied: deepspeed in /root/miniconda3/lib/python3.12/site-packages (0.16.2)\n",
"Requirement already satisfied: peft in /root/miniconda3/lib/python3.12/site-packages (0.14.0)\n",
"Collecting datasets\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d7/84/0df6c5981f5fc722381662ff8cfbdf8aad64bec875f75d80b55bfef394ce/datasets-3.2.0-py3-none-any.whl (480 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m480.6/480.6 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: filelock in /root/miniconda3/lib/python3.12/site-packages (from transformers) (3.14.0)\n",
"Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.27.0)\n",
"Requirement already satisfied: numpy>=1.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (1.26.4)\n",
"Requirement already satisfied: packaging>=20.0 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (23.2)\n",
"Requirement already satisfied: pyyaml>=5.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (6.0.1)\n",
"Requirement already satisfied: regex!=2019.12.17 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2024.11.6)\n",
"Requirement already satisfied: requests in /root/miniconda3/lib/python3.12/site-packages (from transformers) (2.31.0)\n",
"Requirement already satisfied: tokenizers<0.22,>=0.21 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.21.0)\n",
"Requirement already satisfied: safetensors>=0.4.1 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (0.4.5)\n",
"Requirement already satisfied: tqdm>=4.27 in /root/miniconda3/lib/python3.12/site-packages (from transformers) (4.66.2)\n",
"Requirement already satisfied: beautifulsoup4 in /root/miniconda3/lib/python3.12/site-packages (from google) (4.12.3)\n",
"Requirement already satisfied: einops in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (0.8.0)\n",
"Requirement already satisfied: hjson in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (3.1.0)\n",
"Requirement already satisfied: msgpack in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.1.0)\n",
"Requirement already satisfied: ninja in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (1.11.1.3)\n",
"Requirement already satisfied: psutil in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (5.9.8)\n",
"Requirement already satisfied: py-cpuinfo in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (9.0.0)\n",
"Requirement already satisfied: pydantic>=2.0.0 in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.10.4)\n",
"Requirement already satisfied: torch in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (2.3.0+cu121)\n",
"Requirement already satisfied: nvidia-ml-py in /root/miniconda3/lib/python3.12/site-packages (from deepspeed) (12.560.30)\n",
"Requirement already satisfied: accelerate>=0.21.0 in /root/miniconda3/lib/python3.12/site-packages (from peft) (1.2.1)\n",
"Collecting pyarrow>=15.0.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/3a/2e/3b99f8a3d9e0ccae0e961978a0d0089b25fb46ebbcfb5ebae3cca179a5b3/pyarrow-18.1.0-cp312-cp312-manylinux_2_28_x86_64.whl (40.1 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.1/40.1 MB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting dill<0.3.9,>=0.3.0 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/c9/7a/cef76fd8438a42f96db64ddaa85280485a9c395e7df3db8158cfec1eee34/dill-0.3.8-py3-none-any.whl (116 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m53.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting pandas (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/38/f8/d8fddee9ed0d0c0f4a2132c1dfcf0e3e53265055da8df952a53e7eaf178c/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.7/12.7 MB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting requests (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/f9/9b/335f9764261e915ed497fcdeb11df5dfd6f7bf257d4a6a2a686d80da4d54/requests-2.32.3-py3-none-any.whl (64 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m31.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tqdm>=4.27 (from transformers)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting xxhash (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/a7/81dba5010f7e733de88af9555725146fc133be97ce36533867f4c7e75066/xxhash-3.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.4/194.4 kB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multiprocess<0.70.17 (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/0a/7d/a988f258104dcd2ccf1ed40fdc97e26c4ac351eeaf81d76e266c52d84e2f/multiprocess-0.70.16-py312-none-any.whl (146 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m146.7/146.7 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /root/miniconda3/lib/python3.12/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.5.0)\n",
"Collecting aiohttp (from datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/40/7f/6de218084f9b653026bd7063cd8045123a7ba90c25176465f266976d8c82/aiohttp-3.11.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
"\u001b[?25hCollecting aiohappyeyeballs>=2.3.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/b9/74/fbb6559de3607b3300b9be3cc64e97548d55678e44623db17820dbd20002/aiohappyeyeballs-2.4.4-py3-none-any.whl (14 kB)\n",
"Collecting aiosignal>=1.1.2 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/ec/6a/bc7e17a3e87a2985d3e8f4da4cd0f481060eb78fb08596c42be62c90a4d9/aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)\n",
"Requirement already satisfied: attrs>=17.3.0 in /root/miniconda3/lib/python3.12/site-packages (from aiohttp->datasets) (23.2.0)\n",
"Collecting frozenlist>=1.1.1 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/af/f2/64b73a9bb86f5a89fb55450e97cd5c1f84a862d4ff90d9fd1a73ab0f64a5/frozenlist-1.5.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m283.6/283.6 kB\u001b[0m \u001b[31m41.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting multidict<7.0,>=4.5 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/d3/c8/529101d7176fe7dfe1d99604e48d69c5dfdcadb4f06561f465c8ef12b4df/multidict-6.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (131 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m131.0/131.0 kB\u001b[0m \u001b[31m56.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting propcache>=0.2.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1c/07/ebe102777a830bca91bbb93e3479cd34c2ca5d0361b83be9dbd93104865e/propcache-0.2.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (243 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.6/243.6 kB\u001b[0m \u001b[31m41.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting yarl<2.0,>=1.17.0 (from aiohttp->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/1a/e1/a097d5755d3ea8479a42856f51d97eeff7a3a7160593332d98f2709b3580/yarl-1.18.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (336 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m336.9/336.9 kB\u001b[0m \u001b[31m41.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /root/miniconda3/lib/python3.12/site-packages (from huggingface-hub<1.0,>=0.24.0->transformers) (4.12.2)\n",
"Requirement already satisfied: annotated-types>=0.6.0 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (0.7.0)\n",
"Requirement already satisfied: pydantic-core==2.27.2 in /root/miniconda3/lib/python3.12/site-packages (from pydantic>=2.0.0->deepspeed) (2.27.2)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.0.4)\n",
"Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (3.7)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2.1.0)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/lib/python3.12/site-packages (from requests->transformers) (2024.2.2)\n",
"Requirement already satisfied: sympy in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (1.12.1)\n",
"Requirement already satisfied: networkx in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.3)\n",
"Requirement already satisfied: jinja2 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (3.1.4)\n",
"Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (8.9.2.26)\n",
"Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.3.1)\n",
"Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.0.2.54)\n",
"Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (10.3.2.106)\n",
"Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (11.4.5.107)\n",
"Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.0.106)\n",
"Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (2.20.5)\n",
"Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /root/miniconda3/lib/python3.12/site-packages (from torch->deepspeed) (12.1.105)\n",
"Requirement already satisfied: nvidia-nvjitlink-cu12 in /root/miniconda3/lib/python3.12/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->deepspeed) (12.5.40)\n",
"Requirement already satisfied: soupsieve>1.2 in /root/miniconda3/lib/python3.12/site-packages (from beautifulsoup4->google) (2.5)\n",
"Requirement already satisfied: python-dateutil>=2.8.2 in /root/miniconda3/lib/python3.12/site-packages (from pandas->datasets) (2.9.0.post0)\n",
"Collecting pytz>=2020.1 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/11/c3/005fcca25ce078d2cc29fd559379817424e94885510568bc1bc53d7d5846/pytz-2024.2-py2.py3-none-any.whl (508 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m508.0/508.0 kB\u001b[0m \u001b[31m38.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hCollecting tzdata>=2022.7 (from pandas->datasets)\n",
" Downloading http://mirrors.aliyun.com/pypi/packages/a6/ab/7e5f53c3b9d14972843a647d8d7a853969a58aecc7559cb3267302c94774/tzdata-2024.2-py2.py3-none-any.whl (346 kB)\n",
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m346.6/346.6 kB\u001b[0m \u001b[31m36.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
"\u001b[?25hRequirement already satisfied: six>=1.5 in /root/miniconda3/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/lib/python3.12/site-packages (from jinja2->torch->deepspeed) (2.1.5)\n",
"Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /root/miniconda3/lib/python3.12/site-packages (from sympy->torch->deepspeed) (1.3.0)\n",
"Installing collected packages: pytz, xxhash, tzdata, tqdm, requests, pyarrow, propcache, multidict, frozenlist, dill, aiohappyeyeballs, yarl, pandas, multiprocess, aiosignal, aiohttp, datasets\n",
" Attempting uninstall: tqdm\n",
" Found existing installation: tqdm 4.66.2\n",
" Uninstalling tqdm-4.66.2:\n",
" Successfully uninstalled tqdm-4.66.2\n",
" Attempting uninstall: requests\n",
" Found existing installation: requests 2.31.0\n",
" Uninstalling requests-2.31.0:\n",
" Successfully uninstalled requests-2.31.0\n",
"Successfully installed aiohappyeyeballs-2.4.4 aiohttp-3.11.11 aiosignal-1.3.2 datasets-3.2.0 dill-0.3.8 frozenlist-1.5.0 multidict-6.1.0 multiprocess-0.70.16 pandas-2.2.3 propcache-0.2.1 pyarrow-18.1.0 pytz-2024.2 requests-2.32.3 tqdm-4.67.1 tzdata-2024.2 xxhash-3.5.0 yarl-1.18.3\n",
"\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"!pip install transformers sentencepiece google protobuf deepspeed peft datasets "
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "4e906370-40c7-4f6b-a700-f183a9308c78",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"https://hf-mirror.com\n"
]
}
],
"source": [
"import os\n",
"\n",
"# 设置环境变量, autodl专区 其他idc\n",
"os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\n",
"\n",
"# 打印环境变量以确认设置成功\n",
"print(os.environ.get('HF_ENDPOINT'))"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ecc98529-6581-41d2-a876-23ce5187cae7",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "b01fc372-33af-46e5-8c0e-8bccba7237ee",
"metadata": {},
"outputs": [],
"source": [
"from datasets import load_dataset\n",
"# load ~11k samples from promoters prediction dataset\n",
"dataset = load_dataset(\"dnagpt/dna_core_promoter\")['train'].train_test_split(test_size=0.1)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "136c38d4-bd0f-4ecd-9165-2fd5b5207c1d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"DatasetDict({\n",
" train: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 53276\n",
" })\n",
" test: Dataset({\n",
" features: ['sequence', 'label'],\n",
" num_rows: 5920\n",
" })\n",
"})"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dataset"
]
},
{
"cell_type": "markdown",
"id": "28acb64e-8d1e-4482-a515-344a2e0344c4",
"metadata": {},
"source": [
"## lfs 支持\n",
"apt-get update\n",
"\n",
"apt-get install git-lfs\n",
"\n",
"git lfs install"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d3cefb0-1eed-4f23-8591-1990f7113820",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/.ipynb_checkpoints/gene_bpe_seg-checkpoint.vocab
+60000
View File
File diff suppressed because it is too large Load Diff
04-gene-sft/.ipynb_checkpoints/gpt2-small3-1024-checkpoint.json
+602
View File
@@ -0,0 +1,602 @@
[
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCAGGATGCGCTGACGACCCGGCTGGCAGGCGGGTCCTCGTGGGCGAGGCGAGGGAGGCGGCGAGAGAGGAGCAATAGTTTCCCACCGCTCCCTCTCAGGCGCAGGGTCTAGAGAAGCGCGAGGGGATCTAGAGAAGCCGGAGGGGAGGAAGCGCGAGTCCGCGGCCCGCCCCGTTGCGTCCCACCCACCGCGTCCCCTCCCCTCCCCTCCCGCTGCGGGAAAAGCGGCCGCGGGCGGCGGCGCCCACTGTGGGGCGGGCGGAGCGCCGCGGGAGGCGGACGAGATGCGAGCGCGGCCGC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGTCATGCCTGCAGGGAGAGAGGGCTGGGGCGCTATCTATGGTAGAAGTCAGAAGAGGCTGTCTATATGTCAGTGGGAGGAGAAAAGTGGCTCCAAAGCGGACTGTCTGTCCCTGTGCCAGAAGACATTGCTATTCAACGTTCACTCGCTTCTGCACAGGCCCACGTCACCCACAGACCGAGAACTCATCCACTAAGCGCTGCGGGTGGGCAGTACATACCGACGAAGTGCCTTCGCTACCCCTCTGGGTGTCCGTGCTCCCGCGAACCGGCTGGGGCTGCAGACGGGAAAGGAATGTCC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGGGTTTGCACTAACCGATCCACTCCGCGGGGACCTCTTAGAATTCTGCTGTGGATAAGGATATCGAACTACCGAGTCACTGGACCTAAAGAGAGATTCACTGTATCATCGTCTGCCTTAGACTGTTCCGCACCCAGGAACTGGGGCTGTTGCGCCCTTAAGGTTACTTTGAAGCCAAGGTCGCAAACAGACTTCCGCATTGACGTCAGTAGCCGAACGCTGATTTTCTTAATCTAGTATTTAGGATGGGCCTCTGTCGCCTAGCCGCTATCGCAGAGTGGAGCGGGGCTGGGAGCAAAG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCTCGTAAGGACCTAACTGTTCCAGCACTACACAGCTCATGGTGTCCTCAAGATTAGTACATATCAGAGGCCATGAACTTAGTCTTACGCTCACTGTGGCACTGGACGCGCGGGGGATGCGGGGTTGTTCCCAGCGATTTATGATTCGTTATTTGCCGCGTTGACTCGCCGTCTGTAGCCCCATGACAACGACATTCCTGCATTCTCTGCCTGGAGAGCGAAGTGACAATACTGAATTGAAACGGCTAGAATGTCGCTCGCTGAGGCTCCGGACCTTGGAGCGTCTAGAGTCTGGCTAC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TGCCACATAAGTCGGGTCAGCAAGTCGGTGGCGAAAAGAGGGCCACGTCCCAGACTTTCTGGGAGGAGGCGGGACAAAGGGGCGGGGCGAGCGCAGCGCCCTCCGGGTGGGCGCCGTCAAGAGGCCGGGGGCGGGGCCGAGCGCGGCTGGGCGGGGCCTTGAGAGGCCGGCCGGGGGCGGGGAGGCTGGCGGGTCGGCGCGGGCCCAGCCGTGCGTGCTCACGTGACGGGTCCGCGAGGCCCAGCTCGCGCAGTCGTTCGGGTGAGCGAAGATGGCGGCCGAGAGGGAACCTCCTCCGCT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ATAGGATGCGTCTGTACTCTGATTAACAAACTGCTGAAGGAATAAATATGTACTTGCTGGGCAGCGCCGCCGGCCGAATGGAGATAAGCCTATGCAGCTTCGTGCGCGGCTCCCCAGCCCTTTGCTGCGCCGCGAGCTGCGCCCTGAGACCCCCGCCTCGCTGCCAGCTACTTACCTGCCCCGGCGGAGGGGGCCATGTTGCTACACCTAGGCAGGCGGCAAGAAAGCACGCGTAATGAATTCCTTATATCCCCCGCGCCCCAACGGCGGCGGCGCGCCGGCCGGCATGGAGCCCCGCGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGGGTAAAAAAGGGGACAAATGGAAATTTGTGTTGCAGATAGATGAGGAGCCAACAAAAAAGAGCCTCAGGATCCAGCACACATTATCACAAACTTAGTGTCCATCCATCACTGCTGACCCTCTCCGGACCTGACTCCACCCCTGAGGGACACAGGTCAGCCTTGACCAATGACTTTTAAGTACCATGGAGAACAGGGGGCCAGAACTTCGGCAGTAAAGAATAAAAGGCCAGACAGAGAGGCAGCAGCACATATCTGCTTCCGACACAGCTGCAATCACTAGCAAGCTCTCAGGCCTGG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCTGGGTCCTAGAGGCTACTGAGACAAATGCTGGCTGAAGCCGAGGTACCTAGATTTGTGCAAATCGAGGAGTTCTATTCTGCGTGTTTGTGAAAGGGGGTTAGACGAGAGGAGTGCGGTGGAGGCGAAGATAATAGGATTATTCTTACAGTGTAATAATAATTATTGGTGCTAATGTTTAGGAAGCCGTGAAAGATAACACGAGGGGAGAAAGTGGATAGAAAAACCACCGCTGGCAGTTCTTTTCATTTGAAAGTGAATTGGGAAAAAGCGAATAAAAATTAGTCATGAGTTTTGGTA",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CTCGTTGCGTAATCCTCCTTCCGCACCAACGCAAGGCAGGACAGGAGGTCAAATGAGTTCGAGATCAGGCGCCACAGACTCCACTAAAGCACAGACACCAGGTGCGGTAAGACACGCTGGCTCCAAAGTAGGACACCAAACCATTGCTAATGCCTGGGGAGGGCCAAACATGGGCAGGCTGGAGCACTCTCCTCAAAGAGGACCACAAGCCGTGACAGCAGAGCCCCTCGGACTGATGGGACACGGCTTAAACTGAAGTTCGAGGACTCTGTTCTGCAGAGGTCTGAGGCTGGAAGGGGG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CGGGAGAGAGCTGGAGTCTCGCGGCCTCGGAGGCCGGGAAGCCAGCCCCGGCGGGGACCCGGGCGGGGCGGAGCCAGCGGAGGCCCCACCCCCGGCGTCACCGGCCCCCAGGGGGGCGTCGCCCCCACCCCGCGCTCCAGGTAGCGGCGGCCTCAGCTGCCGGCGGTGGCTGGCTGAGCTCCCCAGCGGCCCGCGGGCCGGGGCGGGGGCGGGCCGGGGGCGTTCCCGCGGGCCACCGCCCGTGATGTCACAATCGCGGCGGGCCGCGGCGCTCCGGGGTCGGCGGTGGGCGGCGGGCCG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGACTATACCACTTGCCGGCCGCGCAGAGCCCTCCCCAGGTCGCGCAAGGAGCTCGAGCTACTAACGGAGGAGCGGCGCAGGCCCCGCCCTTAGTTCATGAGCTTACTTGTACAGATCTCGGCCCCGGCCCCGGCCCCGCCCCCTGCCTCATCGTGGACCATCACGCTTCCAGCCCAGTCGGCTACGGAGATAATCCATGGAGCTCCGAGGCGCCGAGAGAGCAACCTTAAAGTATCGGGATCAGCGACATTATCAGCGCTGGTCCCTGCAGCTTCTGGTTCGAGATCTGGAAGGGCGGT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AAGTTTAAAAATGAACCCCCAACAGAGCCACCCGCCGGGGACGGTTCTGTTTTCAGCACAACCACGGGGTTCGTTTAGCCTGTCGCCCAAGTTTGTTTCCTCTAGGTCGCCCACCTTCCGTCGAATTCTGTCACTAGACTTTTATACTGGGTTTGTATGTAGCTCGATTCTGCATACAAGCTGGACTGAGAACTGGCAGGAGGCGAAGATGAGGGGGCGGCATCCAATTTGTTGTGATCCTTTTGCGTCAGGCTTCTGCCTGAGCTCGGTGAGGTCAAGCCTCCTCTGCTTCCACCCCTC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGTCCGCTCCGCGGGGCAGGCGTCGTGGGCCCAAGGAGGCCAGGCCAGGAGTCTCGGCCTCGGGGTTACGCGGCGGCGGGCAGCTCTTCCGCTCCGCTCAGGGTGCTGCTAACCCACAGCCAAGCAGCGACCCGCAGGACCGGCGAGAACCCCGCCACTGACTCCGGCCGGAAGAGAACGTGCAGAGATGTACGTCACTTCCGGGGCGTGGCGTGGCCGGGCGTGGCCGTCCCGCCCCTGGCGGCGTGCCTGGTGCCCTGTTCCGCGTCTGTGCGACCGTCCGTCCCGAGCGCGCAGCCG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TACCGGTGCTACTTTACGCTGCGTGTTCCCTAGCAGGGGACCTAGTGTGCTACCATAAAAGCACGACGATAAGACTATACCTCTTCGCCTGCTACTAACATAGGGCACTTCTTCCCCCTCAATCTATCGACGTTCGTTAATATCACCGACTCCCAGGGAGGATCTAATAAAAGGAGGAGGGCTCCTGGAATAATAGAGGAGTGGTGCTGCGAGGGGAAGGAAAAGCCAATCCGTGACCCCTTTCCCGGAAAGCTTCGAACCGACAGCCAGAAGCCCGCACGCACCTCGCACCATGAGATG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGTCGAGAAGACCTGGGCTACTGAATGACGAATGGGACGCACACGTACTGGCTACTTTTGTGGAAGATGGTCCGCCCTGGTTTGGATTGCGCTTTGCCACGAGAACCGCCGGGAAGGTGGGGGACGGGGACGGGAGCAGCCAATCACGCGCCGTCCCGTTCGCTTCGTCAGGGCGCGATCGCGGATTCGGCACTCAGTGTATGGTACTTCGGGCGAGCGAGGAGTCGCGAGCGCTGTGAGTTTCTCCTGGAGTTTTCTGCCGTTTTCGGACCCTAAAGCTTATCTAGGCTCGCGCTCTGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCCGTCCTGCTCCAATTGCCAGGAGATACCATTAGCGGACTGACTCCAAGAAAAGAGATCCCTTGAAAAAGGCAGTGCTGAACGCGAGTATGGCGCTGCTTGCGGGCCATTCTGTGCGCTACGCTACCCGCTCGTATTAATCAACAACAGGACCGGACTCTGGTTGCTTGGGAGTTCAGGTGGACAACAGCAACTACTAACAAGGTTTTTCGTTCATAGTGCGGGAATGGCAAAACACGAGAGAAACGCACTCTTCATAGGTCACTATCCGTCTGGAAGAAGGCTTTGGACAGAACAGGG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCCCTGCCCCCAGGCACTGACAGGGTAGCAGAGGGAAGACACATGAGCTGCTTTTTAATTTTTTTTAGAATTAATAGAGACGGGGTCTCACTCTGTCACTATGTTGCCCAGGCTAGTTTTGAACTCCTGGCCGCAAGCAGTCCTCCTACCTCAGCCTCCCAAAATGCTGGGATTACAGGTGTGAGCCACCGTGCTTAGCCATGAATTGATTTTAAAACCTTAAATGAGGGGCCAGGTGTTGTCTTTCCGTTTGCCTGAGTGACGCGGGTCTTCTCCAGGACATCCGGCCACAGATCTGCC",
"output": "Non-promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CAGCCCCTATGTTGGTGTGGAGACCCTCACCCGAGCTGGGCCTGCACCCTACACCAATCCGAGGTACCACGATCGGATTACTACAACAGCGGGGTCAGCTCAGGCGCTCGGTCCCCCGGGATCCTGCTCCCAGGCACACGATTCCGTTCGTTATAATTATGTAGACGGGGCCTGCGGTAGACCGACTGCGGGTGACCGAGTAGTGCAAAGTCTGGCTTCTCTTTTGATAGAGCACATCGATCTGTCTTCCTCTTTTCACCGTCGATGAGCCCCGCGTGCCTGGGACTGCTCCAGGCTCCA",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GTCGCCGCCCACATTAGTGTGGGGCCCTGCGGCCTAGCGTCCCTCACCAGAGGCCTCCCCTTGCCTAGCTGGACCGCCGAGGGACATCGACGAGTATCCTCCTCCTGCTGTCCCCGGCTTCGCCTGCCGCCCCTAACCGGCCAGTCAAGATGGCCGCCGCTGGGTGAGGCAAGCTGGCGCGCCGCGGGGGCGTCTGGGAGTTGTAGTTCGGGACGGCGGGCTGACGCACTTCGCCGCCGGCCGACGGGCGCCATTGTGCGGCGCGCGCCGGGTGAGTGCCGCGCGAAACCTGCGTCCGTC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCGGGTAGCGATGGGCCGGGCCTCAGTTGTTTAAAGGCCCGATGCGACCCCGCTTAAGACGTCAGATCGTTCACCCCCGAGGGTCGCCGGATTTCGCTCCCGCGCGAGATGGCCTCCACCAGAGACTGCTGCTGCGCCTTGGAGCGGTGCCTCGCCTCCCAATCCTCGTCGTGAATTGCCCCAATTGCTGATCTGGGAACAAGCTCGTGCCAGGCGGGCGGAGAGTACGAACTCCTTATAGACCAGGATGGGCGTAGCAAGAGCGCGGGCGGCCCATGAATGTCTGCAGCGGTGAGGACC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CGGATGCGGGGCCGGAGGGGATGGAAGGGGCCGATCTAACACGATATTCCGTTCAAGAACTACAACCGTCTACAGCACACCATAGCCGCTCGCTGAAGGATGTCGAAGGGCGCCCAGGCACCCACCACGTTTGGCCCCTGCCCGGGCCGGTTCCGATATGACTCACTCATTTACTCCCCTAGAGCTCCGAGCCTCGGATCGCCGAGGTGACCCTTACCCGCCTTCCTACCACCTTAGGGGATTTGGCCCGGAGAACGAGATCACCCTCTCATCGTGGCGGGGTATTCCCTTTAAGGTTTG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCCCGTCGGTCCGCGGGGCTCTCGCTGGAGCCTCCCCGACTCCGGTTTCCCCTCGTCCAGAAGCCCGACGTAACCAAAGCCCAGTCTGTCACTTTAAACACGCCCCGCCCCGCCTCCCGCGGCTGTGTTGCCTCCTCGCTGGAGAACACCCTGGTCGACCTCTGTGCGTCCGTGTGCGCGAGCGCGTCCCGCCGAGGCGGTGGGCAGGGCGGACGGTGCGCAGTGCGTTCCCGCTGGTCGGAGCCAGCACACTAACCACGCCACGCGCCCTGCCGTCCCTTCGCCTCCAGCCGCTGCAG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CACCTACTCGGGGTCCGAGGAGGGAGGATTGCCTAGGCCCAGGACTTTGAGAACACGCCTCTACAGAGATTTACATTTTAAGAAAATTAGCTGAGTTCGGGTCGGGCGCAGTGGCTCACGCCTGTAATCCCAGCACTTTGGGAGGCCGAGGCTGGCGGATCACCTGAGGTCAGGAATTCGAGACCAGCCTGACCAAAACGGCGAAACCCGTTCTCTACTAAAAATACAAACACTAGCCGGGCGTGGTGGCAGTCGCCTGTAATCCCAGCTACTCCAGAAGCTGAGGCAGGAGAATCGCTT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCCGGCCGGTGTGCGGGACACACTATTGTAATCCTAAAAGGCCATGGGCCCCTGCTGGGGTCAAGAATAATGCTCAAAGTTTAAACTACCGAGTGTTGCTGAGGCTTGGGTTGCACATCGAGAGTCGCGATAACACTCACCCAGGGGAACATACCGCGAGGGATTACGGACGATGGAGAGATCACCCAAGGCAGTGGGAGTCCGCGCTTCCTCCCTTAAGTACCGCTGGTTGTCCCCCGACGTAGTGTAGATAGTGAGCCCTCTTGTCCGCTGGCACGTGATGTATGTGTGTCCTTGTCT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGGTTAGGCGCGTGCCGCGAGAACAGAGTGGACGGAGCGTAGGAGAGACCGAAAAGGCTGGGGGTGGGAGTAGCGGATTTGAAGCACTTGTTGGCCTACAGAGGTGTGGCAAGCAGAGCACCTCAGAACTCAGGCGTACTGCCCGCCGCCCGAGCCCTGCGAGGGCCGATAGCGAGGGTGTGGCCCTTATCTGCACCCAGCAGAGCGCCGGCGGGGTACGGTCTTAGGACCTCGATCTCCTTCTCCCTCATTTTCTCTCATCCCTACCTATTGTGGGTGAGTCCTGGCCCCTGGACGGGG",
"output": "promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CTTGCATTAATACACCCGGCGCCAGAGGAACGTGGGGATCAATCATATTACACTGCTTCAGACTTGTAATTTTATTTGGTTCAGGCATCGTAGTGGTGAGGCTTGGCAGAGAGCCCGAGATGGAGCTATTGAGACGTTTCAGGCTGGGCGGATAATCGCATCAGTAGGGACTTGGCTAGATCTCCCAAGCCTTCTCCACTCTTCAAACTTTCACCAAACTCTTAGGACCCGTCGGGTCGGTTTCATAGACTCAGCCGCTCAACCTTTTAAGCGAGCGGCGCGGCGGGGCGGGCGAACTCG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGAGCGTCATCTTATCATTCTCTCTTTTTTTTTTTTTTCTGGTAATGATTTTTTTTGTTTTTTTTTTTTTTGATATTGACAAAAGTTTAATCATTTCAATTAAAAATGCCACTAATTTGACTTTTTAAGTAAAAAATGTAGGGGGTTTTAAAACTACTTTCCTACTACCAAAAAATCAGAAAGTATCTAGCTTTCTAAATTGGGAAAGCAAGCAATGTTATAAAAACACTGAAGGAATCTCTTTCTTCGTGACCTTTTGTTAAACTCGGTTTAAGCTGTAGACCTTATTTAAAATAAAAT",
"output": "Non-promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGCTGTGAAACAGGGCCACCTGCAACCGTGGCTTTTCAAAGCTGAAGCCATACCCTGGAGGGAGAGAGGATCCTTGGGCACTGACCCTGGCTGTCCACAGCTAAATGATCATGGGACACCCATGACCATGGGCGTGGCTGGAGGAGGCCCTGCCCTCAGGGGTCTATTGGCATCCAGGGGAAGTGTCTACGGGGTAATGATTAACCCACACCTGCCACCTGGTTTTCAGCCTTCTTGCCCTGCTGACTCATTTTGCAAATCCCACTTGCAAAATTTGGGAGCAGCCTGGGTCCAGCCCCC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CTGTGACAATCATGGAGCAGTTGATTGACCCCTTTTCGAGCGCTCGCTATGCTTTAATTTCAAGACCCGCTCTCTAAGGTGCTCTTCCCCTTTAAGACTCACGGCTCTTCCCGTCAGGCTAGCTCGACCTGAGGATTCGTCAACCTCCTTGACAGCTTCCGCTGGCCTGCACCCGAGATTGGGGACGGTTTTCCGCACATCGTGCCTTCATTAGTTAGGTCCTGCGAGTGCATCCAAGGGCCCTCCCCCAAGCCAACAGCAGAACCAGCTCAATTTCTACCCTCACGGCGCTCCGCGGTG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TAGAGAGGCCAGAATCCCTCGGGAGTGCTACCCGGTCAAAAAATCGCGCCGCCCCGTGGGGGACCGAACTTAGCGATGACGGTTCGTCTAATCCCGTAGGAGAGTATCGGGCGAGCTCGGATTGCGAGATTACCTAGTGTTAGTGCGACCGCCCAGGCACATCTGCAGACTGACGTCAGTATGCAAATCAGAGTCGGAAAAGCTGCCAAAATTTTCTCCTGCTGCGCCACCTAACCTGCGTTCGGGGGTAAACCCATCTAGGGCTACTTCAACAAAACTTTGCTGCCCTTCCTGCTCCTC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AAGCTTTGTTTAGGTCCGGAAGGCGGGCTTTCCTGGGAGTGGGTGGGGAGGGGGCGTTGATTCTTGACCAATCCTTTCAGTCCGTTGGGTGGTGACCAGCCAATGGGCCGGATGGATAGGACGCTCCTCCCGGAGAGTAGTGAGACCCCTGGTGCGGGGCGATTGGCGGCGGGAGCGATGAGTGGCAGCCGCACGGCCCAACGGGAGCTGTGCGTGGGCCGCGGGGCGGGGCCAGGGCGGGTGCGCGGCGGCGGCGGGGTGGCTGGGCCGGCGGCGGCGGCGGTACGAGGCGCGCGCTCG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCCATCCAGCGAGAGGGGCAGGTTCCGCATTTTCTCTTCCCCTTTCCCAGCGCTTCCTCCAGCACCCGAAGCCCCAACCCTGCGGGTCAGGAACTCCCTAGTCCCCAAGTCTAGGGATGAGATGGGGGAAGGAGAGCCGTCAGGGTTGACCTGGAGTTTTGTCCGCTCCTCCCCTACAGTGATCCCTCTAGCCTTCTCCAGTCGCCTCCGCCATGTCCGAGGAGCTGGCCCAGGGCCCCAAGGAGAGCCCCCCGGCGCCGCGTGCGGGCCCCAGGGAGGTGTGGAAGAAGGGTGGCCGC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ACATCGTATTGCGCTTACTCGGGTCGTGCAATGCACTGACTAGGAAAGGGCTGCGGATGCCTACGTACATAATGCTGTTGCGCACCTATGCACGCGGGTCTGGTCCTGGGCAAGAACCGCCCCCTCTCCGGGCCTGGTTGCCTTTTGATTTGCAGAACAACGGGCCAGGCCCCTTCCCTCTACCGCTCACAAGCGTACATCGCGACCATACCTGGTCCAATGCGCTACGGACGAGGGCAATTCTCGTAATAGGACCCACAGCAAGTCTTGTATCACCTTTGGCTAGCCTGTGCCGGCTGT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ACGTGACCGACGTCTTCCGCATACCCGGGGCTTCCCTCACTTCCAGGACAACCTGTTGCGCCTCTATCTCCTCCCCTCCCCGCAGTTTCCCCGCCTTGGCCTCTAATAGTAGTGAGAGCACTTTAGGCGGTATAAAGTCTGACGCTAGAATCAATCTAGCCGCGGCGATTGGCCACCTGGGGAATCGTTAGCAAAATGTCACGTTTACTACTTTACGGCGATCTTCCAGGGTCCGGGGACTGTGCCAAGGATCCTACCGGGGCTGGCAGGAGCCTAAGACAAATACGCTAGCGGCGGAGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GAGATTTCACGCCTAGCGGCTCAGGATCGTGTGTGCTACAAAAATAGGGCCGCAGGGCCGTATAGTCGCACGAACATACAGGTCGTCCGATTCAGGCTCGGGGGCTTGGACGGAGATGACGGCCAACTCTCACCCTCACCCCTCATTTAGGCGGGCGCCGGGGGCATCATCGGGGTCCCTGGGCCCTCCCCCTGCTCAGTGCGTAGCTCCGATAGAAAAGCTAGGGCATACGTGCAGAGACGGGTTGTCAGGCAGACACCCTGTCTGTTGGCAGCTAGCAGCTGGCGGACGCGACCCGGA",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AAACCATGCAATCCGGATGTGGCAAACAAAGGGTGGGAGCCCACTTGTCATGGCATTAATTGCACGTCTGCTTAGGGCGAGTTTTAACCGTGGCACACAAAACAGCACGATCGAGGATACGGGAGGTCGCCGCGTGATATAACGTCTACCCATGGTTCTCATCGAACCTCTACCTAGACACTGACCAAACCAACGCTCCCAGTGCGCACGGCCATTTCTCATAGATATCTTGAGCAGGTTGCACCACCAAGATAGTGAGCAGGGCGGTGGGTGGTGCGGAGTGCCGAGCGGCCTCACCCC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GTCGGAGGTCGGCGGCCAGAAGCCAGATTCCCTTGGCCCCAGCTCGTCCACTTCCATCCAGCAGTAGCCAAACAACCGTTGAAAATGGCCTGGCGGCGGCCGGGAAACACCAGAGAACTATCTTCCAACCCCTAGATCCCGCCCATCTCGTTCCGGCGGTGTTTCCGTGGCGACGCTATCCGAAGTGCGGCTGCGCAAGGGTGACGGCGCGCGAGCAAGGGGGAGGGGGTGTTTTGGTTCTAGCCGCTCGCCGTCCTTGCAGGCTCTGCCGTCGGAAAGCCGCTCATTCTCGCTTCCCCT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGTGGTTGTAGGTGGGAAGAATCCTGGGTTCCCATGCTGTCTGAACCAGAGTCATTTGTGGCCCTGCCCTCCCCTGGGACTCAATTTCCCCACCTATAAAATAAGCCCCAGTGTCCGCGAACCCTGGAGGGGCCCGCACCACTGCAGGAGCGGCCGCCGGCGCCAGGGGGCGCCTCCTCGATAACTCGGCGCTCGGCTGGCCAGGCACCGGCGCGTCGGCCGCTCGATTGGTCGGGTCGGGGCCGGCCTGAGCGCCGCGGGCCTGCGCCATTGAGGAGCGGCGGGGAGGAAACGCCGCGC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TAGGAGGCCTGTCTGCACAGCTGTGCTGGGAAGCTGATCCATTCCAACCACCGCCTCATGGCCGCTTACGTGCAGCTCATGCCTGCCCTGGTACAGCGCCGCATCGCAGACTACGAGGCTGCCTCGGCTGTGCCAGGCGTTGCTGCTGAACAGCCTGGGGTCTCTCCATCAGGCAGCTAGCCATACCCAACCCCAGGAAGGAAGGCCTTGGATGGACCCTCAGATTGAAGGACCCGGTGGACCTTGGGGTTGGTGAATCCTAAACAGAGAGAATTCGAGGTTGCCTGAAAGCTGGGTGTC",
"output": "promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TGTTCTAGCTATCTTGAAATACGCAATACTTTGACATTAACACACTGCTATGTTCTCTAATGACATCCCTGGAAACCTCCTAGGGGCAGCCAGATCTTTCATGATAGTGGTTGTCAGTCCTCATATGGAGGGTGGAGGTTTGAAGCAGAGAGCCAAGGGAGGTTTTGTGCACCTATGCTTGTTGTGTTTGTACACAATGACTATGCATACGCTGTGAGTATAAAAGGCTCATTTAATCCTATTGTGTCCCAGGCTTGGTTTGTTTTCAAATCATTACAGCATGAATTTAGAAGGTTTTGT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCCCGAAACCCCTCATCTTGGGGGGCCCACGAGACCTCTGAGACAGGAACTGCGAAATGCTCACGAGATTAGGACACGCGCCAAGGCGGGGGCAGGGAGCTGCGAGCGCTGGGGACGCAGCCGGGCGGCCGCAGAAGCGCCCAGGCCCGCGCGCCACCCCTCTGGCGCCACCGTGGTTGAGCCCGTGACGTTTACACTCATTCATAAAACGCTTGTTATAAAAGCAGTGGCTGCGGCGCCTCGTACTCCAACCGCATCTGCAGCGAGCATCTGAGAAGCCAAGACTGAGCCGGCGGCCGC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGCATCTTAGAGTTTCATACAAGGATCAGATTTGTGCTATCGGAGGCCGCCGATCACCCAAGTAAGCTTCATACGGGGAGGGAGGGAGGGCGCGAGCTTGGGAGTGGAGAGGGAAGGAGCAAACTTTGGCATGTATCACAGGGTAGCTCTGCCCCTGCGCGCCCCCTCCAGATCCGGCCCGCGGCTCCCCTCCCGCGGGACATGCTGCGGGGAAGCACGGCGAACAGCCCCTCCTATGATCACAAATGGCTCACCGCTCTCGGAGCCCGGCCGCCACCGCCTGGTGTGCGAGGCTATCCG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCCTCTGACTCGGTTTCCCCTCTCCCCCGGCTCCGCGGTCGCCCTCCTCCTGGACTGCTTATTTCGTCCTCGGCCACTGCCTCTCGGGCTCGAGCTTTCTGCTTCAGAGCAGGAGAGAAACGGAGCGAAAGCAGTTTCTGTCTCCCCGGGGTCTGACTCGGTCCCCCGCGCTCGGTTCTCTGTCCCCTCCCCCTCCCGCCATTGTTCCCGGCAGGAGGAGTGCGCGGCGCAAACTTGCGAGTTCCCCCAGTCTTTGCCCCCGCAGGGCCGCGGACGGAGGTGGTGGCGGAGTTCCCGCT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGCATGATGTTGGCCGACTTGTGCTAAGGTCCTTGGAGAGGTGGAGGCTCTACGGGGGGTGGGCAGTACCCCTTAGAAAACACCCCTCACTCTCAACAACAGAGGCGGTCTGACGTATTGGAACTCATTCTTTAAGGCCAGAGCTGTGGTTGCTCCAGCTGCAATGTGCGCTCTGATGATACTTTATAATAGATTTGAAAAGGGGAGGGACAAGCCTAAGGTCATGGCAGGAGGCTGGGCCAGGGAAGTGTGGGATCCCGTGGTTGTAGTCGGCGAACCGAACGTCTCGGCCTCTCTCCC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TGGGGGTTTTTTGGGGGCTCGGGCGTCGACTGTGGGGCGCTGCGGAAGGGTGGGTCGTCCGCGTGGGGTCCTGGAGCGCAGGAGGCGGAGGGGAAGCCGCGCAGGGGCCACTAGGGAGGGGCTCTGGCGGGGTCGCAGGGGCGCGGTCTGCGGGCGTGGGGGCGGGACCTCGGGGGCGGGGCCTGAGGGCGCGCGCACGCCTCCGGCGCGCCCCCTCCCGGCCGCCATGTTGGCTGGTGTGTGGGTGTCAAACTGAGCCAGACGCGGCGGTGGCGGCGGCTCCGCGGGCTACGGTCGCTC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGAGAGGGGCGGTACGCACCACGGGGGAAGCCAATGAGAAAATCAGGCCCAGCCCTAGGGGGCGGTGCTGTCGGTCACATGCGCACCTGGGGCGGGTGGTGGCGGCGGCGCGGGCACCGCGAGCCGGCGGAAGGGAGAGGGGCCGGCCTGGGGCGGGGTTAGGCAGGTGAGTGACAGGCTCCAGGGGGCCGGCCCCTGCCTGGTGCCCCGAGCGAGCCGGGAGTAGCTGCGGCGGTGCCCGCCCCCTCTCTCCGCCCCTCCAGCGGAGCTGGTCTCCGGCCGGGCACCGTCGCGGGCCCC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CGACAACCCGCCCTTGTTCCCGGTAGAGGAACAAAGAGTGCTGATTTACGAATCTAGGCTGCCAGATTCTGTGTTTCCACTGGAGGGCGCGTCCGATGCAGATGTTGGCTCAAATTCCATCTTAACCTATAAACTCAGTTCTAGCGAATACTTCGGGCTAGATGTGAAAATAAACAGTGATGACAATAAACAAATTGGGCTCTTATTAAAGAAATCCTTGGACAGAGAGGAAGCTCCTGCACACAACTTATTCCTGACAGCCACAGATGGGGGCAAACCTGAGCTCACAGGCACTGTTCA",
"output": "promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TTCCAGCACTCCGGGAGCCCGATTGCGAGTTTGTGAGAGTCCTTGGTGGAGGAGGTTGGAAAGCGTTCTCCCTCCCACAAAACCCGGTGCAGAGGGAGGGAGAGCCAGGTCCGTAGCGTGGGTCCAAGAAACTATGGATCCTGGAAGTGAGCACACTTTTGTGTCTTAAAAAGTTTGGTGTCTCGTTTTGAGGTGAAAAATGAATGAGATAAGTGCTTTTTAAAAAGGTAAAGTTTGTCTTACAACTGAAAAAGCTTTTAAACAAAACTATAGTTTTTGCGAGTAACTCCCGTGACACCT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ATCAGCCCTGGCTAGGACCAGCTTCCCACTAAATCGCTTGCGGCAAGGAAGTCTCAGCCCCCGAGGCAGTCCTCACTGTGAGCAAACCAAAAGATGGGATCACTTGGGGCGCGTACGGGCTCCCAGCGCTGGCAGAAACTGTAGCGTTGTAGTAGCCTTTGTGACTGGTTGCCTGTTGGAAGTCTACCCCACCCCCAAACCATAAATCCACCGGAAACTCCTCCAGGACACCCTCTCAAGCAAGCCCCTGGGATGTAACTGCCACAGGGGCTGCGCCACGATGTTCCCTGGCTCTCAGGT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CACCGCGTGAGGATGTGCCGGGTGGTCCTTTCCTCCTCCTCTTCCTCCTCCTCCCGGCTCCCTGCCTAGTCTCCATATAAAAGCGGCGCCGCCTCCCCGCCCTCTCTCACTCCCCGCTCCTCTCCGCCGCGCACTCTCCGCGGCGCTGGGAGAGGGCGGAGGGGGAGGCGGCGCGCGGCGCCAGAGGAGGGGGGACGCAGGGGGCGGAGCGGAGACAGTACCTTCGGAGATAATCCTTTCTCCTGCCGCAGTGGAGAGGAGCGGCCGGAGCGAGACACTTCGCCGAGGCACAGCAGCCGG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCAGGAGTCGGTAGGGGCAAGAACGTTTCCGGCTCCTCAACGGGAGAAGCGCGGGAAATTTGTTGACGTGGAATAGCTCGACGAAATCATCGGCAGGCGCCCGCCAGGGAGCTAGCAACGGAGCATATGCGACCGTCGGTTCGCAAGAGACACACCTCGGGCCTTGCCCCGGAACGCCGCAATGGCTCCTACCTTTCCGCCGAGTGCAGGTTGATGCTGCACACCCAGAACACTTTGCGGCACGGGGGGGGTTATATGACATGCAAAGCCCTGATTCAACTTGGAGATGAGCGGGTCTGT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GAACTCCTGAACTCAGGTGATCCGCTTGCCTCGGCCTCCCAAAATGCTGGGATTACAGGCGTGAGCCACTGCGCCGGCATCCTGGATGCAACACGATCAACGAATGATTGCCCTTTATTCTGCCTGGTCTCCCAGCGGTGCTGTTGTGTCTTCATTTTAATGGATTTTAATTGATGTCACTTAGGGGGGCCCTCAAGCCTTCCTTCCTCTATCCATTTACGTGGTGTACCATCGAACTTTATCGACTCAGAAGTCGTGGGCGGAGGGCGTTCCGATAGAACCAATTGGCTCACGTCTATA",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AAGATGGCCTGAGTCAGGAGCAGGGGCAGAGCTCAGGTTGTCGTGGGGTCCCGCTGCCCACGTCAGACTGGAGGTGAGGGATGGGCGGGGCCTGACAGCAGGCCTGGAAGGAACAGGATGTCTATGCTGGAGATAGAGGGAGAGGACAGTGCCAAAACCCAGCTCCTGGCCAGTCCCCAGCTCCTCCCTGCCTGGCCCTATCCCAGGATCCCCTCCCCGGCCTCCCAGCTATGATCTACCCCGGGGCCCAGACTTCAGGCGCCTTCACGATGCCGGCGGTCAGTGGTCCAGGTCCCTTAT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCTGACACCCTTCGGCCATTTTTTTTAAACGTGTTGGGGTTTGGGTTTCAGCCACAGGCCTAGCACATGGAACCAAAACGCTACCACGGACTATGGAGTTGTGAACCGCGGAACAGCCGGGCGGACTCCGGGGCTCCTGTGTTTGAGATACGCGTGCCATGAACCTCCGCGCCGGAGGCGAGCCACTGGTAATTTACTTTCCAGTTGCGCCAATCCGAGTCTACCTTGATATTGAAAGGTAAGTCAAGAACAAAATTATCGCATAAGACTGTTGAGGTTACCCCGCGGCGGGGTCGGGGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CTGAGGCTGGCAGGCACTGGGCAGGCAATGAGTTCTTTCCCTGTAAGTTGGGCACAAAAAGATCGGCATGGGAGCAGCCTCCTTTGAGACAGCTGCTCTGAGAGAATGCAATAAGCAGGGAGCAGCCAGCAATTCCTCCTAGCAGAGGGCGACTCGTGGGAGGAGTTCAGTTTGCCAAGTATTGTCATTTGTTGAGAGAAGGTGTGTGCTCAAGGAGGAGTTTTAACCTGGAGGATCATTAACTCTTTTAGTCAGCTGAGGAGCTGCGGTGGCTCGGCGAGTTGGAGTTCATCCTGGAAG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCGCGCCCCTCCCCCCGGGTCGCGCGCCGCGGGCCCGAGCGCGAGACGCCGCTGCTCCCGCCCCCGCCGCCTCCTGGAGTCGCCGGGGCGGACGCGCAGTCCATGGGGCGCGGTGGGCCGGGGAGTTGCCCCAGGGGCCGCGGGAGTTGCTGAGAGGAGACAGGGTTGGGCTTTCTCCTCGCCCAGACCCCACCCCACCCTGCCGACCCCACCCCCTGCTCCTTCCTCCCCGGGGGCGCGCACTCGGGCACGCGCTCGGAAGTCGGGGGTCGGCGCGGAGTGCAGGCTGCTCCCGGGGTA",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TATCTCCCCTCCTCGGTCCCTCTCCCTCCCTTCCCCCTCCCTTCCCCCGCCCTCCCCCTCTTTTTGCTCCTGCTCCCCCCCCCCACCCCGCCTTTCTCCTTTTGCAAGAAAATAATTTGACAGTCGATTTGCTGACAAGGGAGGAATTTGCATCCTGGATTTAAAAAAAAAAAAAAGGCCGAGAGGAGCTTGGGAACGGTTGCTAGGGGTGGGTAATGGGTGAAAAAAGGGGGGTACCGGGGAGCGGATAAGGAGGGTTAAGGGAGGGGGCGAGGATGGGGAGCAATGCAAAGGTAAGGC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GACCAGTAAGTCGGGGGGGAGGGAGCCGGGGCCTGGCCCCGCCCTATGGGGAGGCACGGGTGGCGAGGTCGGCCGTCCCATGCCCCGCCTGCTCCAGGTGCCGCCGTCCCCACAGGTGCCCGCCCCAGGCCCGCTGGAGCAGCCTGTGGCACGTGGGGTAAGTGGAGGCCGCGGCGGAGCCGTGCGTCCGACGGTTCTGGGGCGGGGGTCACCGGGGCCAGGCCCGCCAGGCCCTTACCTCACCAGGCCGCCTCCGCTGTCGCCCAGTCCCGGCCGCTGGCGGGAACTGACCTGGAGCAA",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCTACCGCTGCCGGAAGAGCGCGGCGCCCGACGGAGCCGTGTGGAGGCCAAAACTCCTCCCGGAAGCCGCTACTGGCCCCGCTTGCCAGGCCCAGCGTCTTTTCTGCATAGGACCCGGGGGAAGCCGGGAAGCCGTTAGGGGGCGGGGCAAGCGGGCGGGCGTGCGTCGGTCAAGTTTCCCGGGTCGCGTCAGGCTTCTTTCTCAGCACCGAGCGAGCGACGTTCGTGAAGCTTTCGTTTTGAGCGGCCAGACTCTGCTGTTCTCAAGCCTGGGAGGCACAGAGAAGCAAGACTTCCGCA",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GAGAATCCCTCTATAATATCCTTGCACAGTTCCTCCGCAAGCACAATATGATTAATTTTTACTTGGCTGATGCCGAGATCTTAACTGCCCTAATCCTCGGCTCGAGTCGAGTTGGGGTTTCCTAAATGAATTACTATTGCAAATCTGGTTATGGCGCTCCACGAGATTATGGCAAAGGCTGAAGAATTTATTGACAGATGTGTTGTTAGACTGTGCTACGTTTTAGTGGCAGGTAGTGGAGCAGCCACCGGCGGACTCGAGAACGCAGTAACATGGCTCAAAAATGGTAGTATTTGTTTT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CGGGAACGCACCTGCGTCTAGACGCTGACGCCGTGCGCGGGGCGGGGGCCGGCGAGCATGCGCAGTGTGGTGTGGACAGGGCGTGGTTGGCAAGCATGCGCAATGTGGGCCGGGCGGGGCTGGCGAGCATGCGCAGTATGGGGCGGGGCGGGGCAGGGCTGGTGAGCAAGCCCGCGGGGAGCGAGCGCGGAGGCGGGCGTTGCTACGGCAACGGTGGGCGGCCAGGGAGTTGCCTGAGGGAAAAGGGGCAGACGTCCCTGGGTTCCGGTGTTCGCGGAGGAGTCGAGGCACGGAGAGGCT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GTTCTGATTCTCGGTGAAAACGGTAAACATCGGCGTGCTATGTATGGGGATAGCAGGAGCTCTAGCATTTTTGCGCTGGGAAGATTATTCTTCTCTATCCGTCTATGCGAGCCGGCGTTTCGCAGGGGTACCCCCGGGGGCACTGCGTGGGAGGCCAGGAGGTTTTAGAATGGTCACCCGCCGGTTAGCTCCTTGGTAGTATTCTCGATGGACTGCGCGTTTGAGTCTTAAGGGCGTTTTCACGCTTCCGCCTTCTGGAAGCCTGCACTCCGTTTAGAGGAGCCAGAGACCCGTCCCGGT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GTATTGGGGATGCAGGGCTCCGCGGTAGTTGACCTAGCAGGGCTAGTCACATCACGCCATAGGGAGCAGCAAGGGGCGCAGCAGTCAGCACGGCCGCTATTCAGATACGGTAACCAGACGCCGTGCGGCGTGCGCACCTGGCATCCAATGGCTTTGGGGATAGGGGCCGCCTGTGGGTTGTGGGTACGAATTAACGGAGTCGGAGCGGCGAATGAGCAGTCGCAAAGCCGAAGCCGGGCGTGGCGCTTTAGAGCGTCCCAGGCTCTCCTTCGGAAAGATGTCGGACACGGCAGTAGCTGA",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGCAGCGGCCGCGGCGACAGCTCCAGCTCCGGCTCCGGCTCCGGCTCCGGCTCCGGCTCCCGCGCCTGCCCCGCTCGGCCCAGCGCGCCCGGGCTCCGCGCCCCGACCCCGCCGCCGCGCCTGCCGGGGGCCTCGGGCGCCCCCGCCGCCCGCCTCACGCTGAAGTTCCTGGCCGTGCTGCTGGCCGCGGGCATGCTGGCGTTCCTCGGTGCCGTCATCTGCATCATCGCCAGCGTGCCCCTGGCGGCCAGCCCGGCGCGGGCGCTGCCCGGCGGCGCCGACAATGCTTCGGTCGCCTCG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CTTTTGAACATCTTGTGCCAGGATTTGACATTAAGCTCCAGTTCATCCGCACACCCCGCGGCCGCTTTCGCAAGGGACCGAGAGGGTGGCGGCCTCGCCCGGAGACTCGCGGCGCTGGGAGCTGCCTGCCCTCGGTGCGGGCTGCAGACCGGGCCGCGAGGCGGGCGCGCGGCGCGCACTAGGACCCAGCAGGGCTCCAGGCCGGGGTGGGAGCCGCGCCGCCGACCCCGGGCGGGCGCCGGGCGCAGGAGCCGGGGTTCCGGCCGCGATCTGCTGCAGCTCGGCCGGGAGACGGCGCGA",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TGTGTGGCCAAAGAGAAGAAATGGGTTGAGACAGCAGGCCTGGCACTTACTTTACCTGGCCCAGTCTTGCCTGACAATTAAAAAAAGACGCTTTAGACTGGGCGCGGTGGCTCACGCCTGTAATCCTAGCACTTTGGGAGGCTGAGGCGGGCGGATCACGAGGCCAGGAGATTGAGACGATCCTGGCTAACACAGTGAAATCCCATCTCCACTAAAAATACAAAAACTTAGCCGGGCATGGTGGCGGGCGCCTGTAGTCCCAGCTACTCGGGCGGCTGAGGCAGGAGAATCGCTTGAACC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCATATAAACAGATTAACTGCATTCCCCAACGAGTAGAAATCTGCCCATCACTTTTGGGGTGGTTTGTTGCACCCACTATCGCCCAAGAAACTTCCCCAAGCCGGGTCATATTACAGGTCTCTAGCGCGCTGGCCGCACTCGCAGCCCACCATTCCCACAGAATGGTGTCATCGTAATCTGAAAAGTTGCCACTGGGCACCTGCGCAGGCTTGGCTGCGCCCTCTCGCGCCGCACGCTCCTTCCGGCGACGCGGATAGTCGTAAAGGTTTGAATACCTGATACAGGGAGAGAGGCCGAG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCTTCACAGGGCACAGCCCCTGCAGAGCATCTTGGTCATTTGGAAGAGGACACGGTATCCCCTCTGGCCAGAGTATGTCAGAGAAGGAAGAGTAGGGCTTTTTTGTTTTGTTTTTTTTTAAAGGTGCTTGCTTGTTTAATGTAAATAATAGAAAGCCTTAATATCTTTTCTGTAACACGGAGTAATATTTTAATGTCATGTTTTGGATGTACATAATATATTTATAACAAAGCAGCAAGAGTCTACTTAACCTTGGCTGCCTCGTGGTGTTTCCTGGCTGGGTGGGGTGGGGGGTATCAA",
"output": "Non-promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TGTATTCAGACCAGAACAAACTCGAGTGCCCCAGCTCCTCTGGCAATTAGGTAGCCAATAGTTGAGATTGCGTAGGCAACTACGTAAAATGCGCCGTGCCGTGCCCCTTTCTGCCACTCTGAGTTCGGTAACATAGCTCCGGTTGCTAGGTACTTAAGGTATCCAGTTCCCCCAGCGATTTGCATGAACCGAGAGGGAGTGTCTTCTGCCAACAGATCAGGCGGGGGATCATGAACTCATCCTCCCAGGGAATGCCGGTCGGGGATCCCCCGCGCAGCTCACAGGCCCTGGGAGTGAGCT",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ACCTCGCTATTCTATGCACAGTTTTGAATCCTGCTTTTAAAATGCGATGTCTGCCTTGTCTTTAAATATTCTGAGGGGGAGGAGAGGAGATGTGTAATGCGACTGTCTATAATTTGAAACCGGACCTCACTATTTAGCGTCTCAGAAAACGCGGGGTTCATTGCCCTGGTCCCCCCCGCCTTCCCCCGCCCCCCGAACCTTCCGGCTGGCTCTCTTGCGGTCCCCGCCTCGGCGCTGATGTGGTCTGGCAGTGGAGATTGGCGCCCGGGCGGAGCACGATGGGCTTCCCCGCGGCAGGCA",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCACGAGCGCCAGAGCCCCGCGCCTCCCCTAGCATATCCGGCGGAGTCCCTGCCCCGCAAGCGCCCACTATACGGCCGGACAACTGTTGTGGGCGTGGATGCAACCACGGGGGAGGGGGCAGGCGTGATCTCAGTCCCGCCCGTGCCCGAGCCCGCGCCCGAGCAGGGACTACATTTCCCTCCCCCCTGGGTCAGGGAGGAGTCTCCTCCTCTATCTATTAAGGATTCTCGCCTCCGCACATGCGTTATCCTTCCAAGGCCGAAAGGATTTGGAGTCCTGTACGCCAGAGGCCGAACTCG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ACCCTGCTGGGGGTACAGAGACGTTTCCGAAACTCAGCGGCTCGGTCGCCCCCAAAGGCCAGGCGGAACACACGCCCACACCGCGGCTCCCTTCCCGAAGTAAGACCGCCGGGCCACGGCCGCCCCCAGGAAGCCCCGCGCCCCGCACCCCACACCCGGGGACACAAACAGGCGCCGGGTAGCCGCGAGGGCCGGCGCGGCGCCTTTAAGAGGCGGCGGGCGGCGCTGCCCCCTGGCGGCCGCCCCGCCGCTTCCTCGCCGCCGCGGGCTCAAGCGGGGCGGCCGGGCCAGCGCGGGGCG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AGCTTCACAGTCAGAACGCGGTACCGTTGTGCCCCAGCTCGACTGCAGCCTCCCCGCCCCGATACACTTGTCCCATCCTGTCCACGGCGCCCTTGTTTTGATTCGCTGCTGTCCCAGAGAGCTTAGTTATTCGTACGGGCAATCTTGACCCTTTCCCAGGACCTCCCCTAATACTGTTCATTGATCCTGGACCGGCAGCGAGACCCCTTCCGGCATCCAGAGGCCCAGTACTTTCAATCTCCTCCCCCAAAATGCTTTCCCTTTGTGGGGACCCTCGACATAGATGGACACCCAGCGGCC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCGTTCTTGGCATCCTGAGAGGGCCAGTTAGTACTGAAGTCCCTTGGCTGCTCAAGGATTGCAGGGATGAGGCAAGTGGAACAGCCTCGGAACCTCCGAAAATGGGCACGCTCCAGGTCCCAGTTTCTATGGCAACCATACCGGCAAATTGGGCTCCGCAATGGTTTCTCCTGGAAAAACCGTGATTTTGGTTCCCGCGGACGTCTCTATGGTTTCGACAGCCTAGAAGGAACAAAACGGCATTTCCGGGAAGATGGCGGCGCACAAGTCAGGTCCGGCACATGTTTCCGCGGAGCGGAC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCTCTCGCGCCGCTCGCGTGACCGGCCGGTGTGTGCGCGAGGCCCCGGCTCCCGGGGCACGGACGGCCGGGCGCGCGCCTCTGCGAGGGGCGTCCGGGTCCGAGTCGGCGGTCCGGGCCGGCGCGAGGTGCGTGCGGGCGGGCCGCGGGGGTCCCGGACGGACACAAGCGCACACACTCCCGGAGGAGCCTTCGAGGCTGCTCTTCCTCGGCCAGACGGAGAGCGGCACTGTCTCCCCGCCCAGCGCTCACTCGCCCCGCGTCTCCCCCCGCGGCGGCTGCTCCTCCTCGGCACCGCCAG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GTCCCTCCTCTGGTAGCCAGAATAAATTTTGCAGTAAGTTAGCAGAGTTGATTCCAAAAAGTCTGCACTGTGGCATCTCAGGAAACCAAACTTATCTGCTTAAGAAATAACTTGCAGAGAGAAAAGGAACTCTGACAAGGCTAATAACAGGGATTTCAAGCTGTGTTTCAGCAAGCCAGTCACAGAGTATTCACGTGTTAATTCACTGGCGTGTTCCGCGGCATAAGCACCCCCTCTCTGCCTTCCCCCAGTTCCAACTGTTGTGCTGCAGCAGATTTGGTCTGAGTCTGGGCAGAGCCC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ATGATGCTGGGGACGGGACACAAACAGGCGTGACAGCAGCGACCAGGCCGGCGCGGCGCCCGTAGACTGAGTCGAGACACCCTGGGCACAGGCGGCCGCCCCGCCTCTGTGATAGTTACTGGGACGTGTGCCCAACGGCCGGGCCAGCGCGGGGCGGCGGCGGGCAGGGGCGGCGAGTGCCTCGACGTTCTAGTGCGCCCTTCTCGGCGGTCTTATGCACTATGGCAGCAAGAGTTTTGAATATGTTGCGGTAAGTTGAAATTATCGAGAATGCCAGGCGAGTTGGCCGCGTGCGCCCCG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CGCGACACGCCGTGCGCCTCCGCGGCTGCGCTACGAAAACGAGTCCCGGAGCGGCCCCGCGCCCGCCGCACCCGGCCCTCGCCCGCCCGAAGACAGGCGCCAAGCTGCCCCGCCGTCTCCCCAGCTAGCGCCCGGCCGCCGCCGCCTCGCGGGCCCCGGGCGGAAGGGGGCGGGGTCCCGATTCGCCCCGCCCCCGCGGAGGGATACGCGGCGCCGCGGCCCAAAACCCCCGGGCGAGGCGGCCGGGGCGGGTGAGGCGCTCCGCCTGCTGCGCGTCTACGCGGTCCCCGCGGGCCTTCC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCCCTAGCCAGGAGGTAGCATCTGTCTCCAGGTGCCCGTGGGGTGTCCTCCATGAGGACCACCTCTTCTAACCACCTGCCCAGCTCAGAGGACCAGCTGGCAGGAAAGACGCTGCACCGCCCATGAGGCCAGAGCTACAAGTTGCTTCTAGGAGTGGCTGTGGGCGGAGGCTGGTGGTTCTGAAGGTGGCGGTGGTTCTGCAGCGTGGCTCCCCACAGCCCCTTTCATCTGAACAGTAGGGCTCATGCCACTTGTTAATCAGCCTTCCAGTGGCACGAGGAGTCATTTTCTGGGGATAGG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCCTCCTCCCACGCTGCGCGCGCACCTCCCCGCCCCCACCCCTACCCGCTGGCGTGCCCAGTGGAACGGAGCCTTGTGTCTCCGCCTCAAGTCCCCGGATGCTCACCTCCCCGACTCGCCCCCGCTGTGGCCCCGCCCCCGCGCGGCTCTTCGTGCCACGTCACCGCCTGCGTCGCTTCCGGAGGCGCAGCGGGCGATGACGTAGAGGGACGTGCCCTCTATATGAGGTTGGGGAGCGGCTGAGTCGGCCTTTTCCGCCCGCTCCCCCCTCCCCCCGAGCGCCGCTCCGGCTGCACCGCG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCATGCAGCGACGGCCGCCGCGGAGCTCCGAGCAGCGGTAGCGCCCCCCTGTAAAGCGGTTCGCTATGCCGGGGCCACTGTGAACCCTGCCGCCTGCCGGAACACTCTTCGCTCCGGACCAGCTCAGCCTCTGATAAGCTGGACTCGGCACGCCCGCAACAAGCACCGAGGAGTTAAGAGAGCCGCAAGCGCAGGGAAGGCCTCCCCGCACGGGTGGGGGAAAGCGGCCGGTGCAGCGCGGGGACAGGCACTCGGGCTGGCACTGGCTGCTAGGGATGTCGTCCTGGATAAGGTGGCATG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCCCTGAGCCAGGCTGAAGCGGGATAGGAAGGGTAAAGGTGCTAGCTGAGCCGTAGTACGGCCCAGCAGTTCTCCAGGGGACAAGCTAGGCCCCCGCTGGAGGGAGTGGTGAGACTCAGCCGGGGCCTGATTCTGGTCAGCACCTTGGACAGCAGCCAGTGCCCTCCCTGGCCCCGCCTGGCCCCGCCTGGCCCCACATCCGCTACTCCTGGGCACCTCCTCAAATAGCACAGCCTCCAGCTGGTGCCCAGGGCCTGAATACACAGAGCGCTGAGAGAGTGGGGCAGTGTGGTCACGGAC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CCTGGGAAAAGCTGAGAGCATAATCGAGCGGCAAAGAAAACGCGACCACTGAGGACACTTAGAAGGGGGCGGGGCCTTTTTGTGGTACTAGGAGCTCGAGAAATTGACTAGATTACGAGTGGTTTACGACTCAAGCATACAACTTACAGCTTGGGCCTTCCGTAGCGGGCGTTGCGGGGGAACGTACAGTGCTTGAGGGGCGAGGTTTGCCGGAGCCTCATTCTAAAGGAATCTTCTAGTTCCAAGCGTTGTTCCAATTGGGCCGGAGCGTGCAACGGGACCCACGGAACTACAGGTGTG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AAAGCCACCTCCGCGCCGGCAGCTCCGTTTCTAGGGAAACAGGACCAATTCCAGAAGTTGGGGTGTTACGTGTGCGATATACCTGCCCAAGAGCTGGTATCAATAGAGCGGGCGGCATGGTAGTGTTTCCCCCGGCCATGGTAAAATGCGAACCAGATCCTCGGCGGGTAACCGCGCCCCTAGGCGTTGGCATGTGTCGGGGTAGCATGACGGGGTCGTCCAGTTCTTAGCCAAGATGAATCCGGCTCCGAAGGTCCCCGGCTTCGTGGCCATAAAGGCGAACACCGCTGGAAAGGTGGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGGCTCGGGGACGGGACGGGCGAGCAGTCATTTCTGAATAGCGCGTGCAGAGGGTCATCGCGAAAAGACTAAGACGCGACATAATGGACTGACTCGGGGCATGGCGGGGTGGAGGCGGGAAAGTGTCGAGGCCGGCTAATGCCTGGACAGGGAGGTGGGCATCGCCAGGGCGACGATGAGGGTCGCCACTCCGGAACACGATTAGCATCAGTGCACTGACCTCTGCCCCCACGACTCTCCAACACGCCCGATGATGTTGCGTGCGGCAGCTGTCGGCTAGGAGTCGCGGGGTCTGTGCGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TATGGTATTTCAAGGTCATAAATTACAAATGTGCATATTTATTATTCCCCTATAAAATGTTAATTAAAAAATTTTCATATTTACTTGATGTAACAGAAATGAAACCTGTAATGAGGGATCTAATTTCTTTTTCTTTTTTCATTCCAGAATGTATGACAACATGTCCACAATGGTGTACATAAAGGAAGACAAGTTGGAGAAGCTTACACAGGATGAAATTATTTCTAAGACAAAGCAAGTAATTCAGGGGCTGGAAGCTTTGAAGAATGAGCACAATTCCATTTTACAAAGTTTGCTGGA",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "ACGACCCTGGCCTCCGACTTCAACGACTTCATAAGGCGGCGTTTCTGGGCGCAGCCGTGTCGCTCCTGGTGAGAGGCCGCCGGCAGGCGGGATCCAGCGCCCTCCGGGGCACCGCGGGCGAGACCGTCGCCTTCGCACCCCCGGCGCGGTCGACCCCGCGGGGCCGTCGGGTCCTGGGTTCCCCGCCGCGTTGCGCTCGTCCCCCTCCTGTCAGAACCTGGGCCCCCGCCCCGCCCACCGGCGCGGGGCCTCTCCTCCTCCCGCTAACGGGCGGTCGGCCGCCTCCTTCCTCTCTTGGCT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGTTTCTGAGGCTGAGGAGGTACCCAGGTAAGCACACTCTGGGTTCTCTGGTGAGCACACACCGAGCTGCCTGCATCTCCCCTACCCATGAAGGCAGCTAGCCGCCGGGCCAGTGGTGGGAAGGGGAGGCGGCCATTAGCGCCGTTCCCGCCGGGACTTGAAGCGCCCGGCCGCGGCAAGCCCCGCCCTCGGCGCGCCCCCGCGTCCGCGCGCGCTCCTCGGGTCTGCGCGGAGCCGGCGTCGGCGCGCGCTTGGGCGCCTGGCGAAGACCGAGAGAGGCTGGCGGGATCTCAGCGGCGC",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GACCGCCCGTTGGACATACCTCTAAATTGGGAGCGCGGTGCCGATTTACGTGACCGCCGTGGTTCGCACGGTGGGTAAACAGTGGATCGTGGGGTGGTGAGAAGGTTCCGATGGGTGTTCTTACTACGGCCCGATCCTAGCAGTCGCCGACTTTGTTATTCTTATTGGTGGATGGGGGGCCGCTGATATAAATCTGGGAGATGAGGAGATAGGCGGGAGCGAGGGCTGACTCTGCGTGTCGAAGGAGGTAGAGAAAAAGCGTTGCTTAAAGGGAGGGAGCGTAGCTTGGTTGCTCCGTAG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GACCTCTCTTCGGGCGGCAAGTGCCATACGTATAGATAAGCGGAATTTACATGGTGTCACAGTTAGCGCAAGGTCGAATAAATGCGGCTTAACACGACAGGTATCTCCAAATAAGTGTGTCTTACGGCGTAGATCGCGAGTTGAGCTGTCATTTCTTTCCAAGCGGCGCAGGACAGGACCCAGAAGCCAGAGCGCAGGAACTGCTTCCGGTTTGAGTTTTTATTAGAATAACGGGCTGTGGGAGAGGCAAGTGGCGGATCTTCAGGTCGGCGAGGGTGTGACATTCGACCCACAGGCCGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCATATTCTGTCTGAAATCGTGTGCACCGAAATCCCCGCCTTGCGGTGGAGGCTGGCGCTAGGCGGCCTCAGCCTCGGCCTGCTGCGCTCAGGAACCCGCGCCCCGGCTCCTCGGCGATCCATTGCTCTTTCCTCTGGCGCCGGCCGCAGGCCTCGGTCACGCCCCCAGCGGCCCGTTGGTTTCCGGGTCCCGCGGGGTGCCCCCGCCCACACGCTATGCCTTAAATTGGGCCAGGCTGAGGCGCTGCTGCTGGAGCGGCCGATCCGAGACGTGGCTCCCTGGGCGGCAGAACCATGTTG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TGGTCTTGTGTCTCGGCGTACCATATCTTTTGGGTTCCCCGGCTGCCCCGGAGGTTCGCGGCGAGAGGACGGGCCGGGTGCGGCTAGTTTACCTGAGCAGTGTGAACTACCTCACTCCCATCGGGACCAGCCGGCATAAACACTCTTATTGCCTGTCAACGTTTTCGAGACATATTTTGTTCGCGAGCCCGCAGACAACGCGCATTCTGGAGTTCGAACGCTCAGTTCGCGTCCCCGAGAAGATATTATTCAGATGCCTGCGAGCTGGGGCCGGGTGGCCGCGCCGGGTCGGAGTGTGGC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCCGGAGACAGGTGGGAGGGTGGGGGATGCCGGCAGGAGAGGGAAGCAGGACCGTGCGCGCCGCCACCCACCCGCAGTCCGTCCGCCCCCGGGCGCCCAGCGCGTGGCCCGAGCGGCGCTGCCCGCCGAGCCGGGAGCCCCCTCGGCCCCTCCCTTCAGGCGCGGGCCGGGGGCGGCTTCTCCGCGACCTTATGTAACCGGGCGGGAGGGGCCGGGCGGGCATGGGCCTTCCCGGCCCGGAGCTGGGAGTCGAAGGGGCGGGAGGCGTGATGGTGAACTCGCAAGAAGTTTGAGGGACG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GCAGGACTGAGTCACCTCCTGAAGGCCCCGTTCCAAAGCACTCCCACGGGGGATTAGGTTTCAACACAGGAATTTTTGGCGACACAAACATTGAGACCACAGCACCCAGCTCTCTGGGTTCCCAGAACCCTGTGGGAGGGCTGGGTCGCGCCTGCCCCCTTTTAAACACGAGGCTTCCATGGAAGCTTCTTGCCGGAGGCTGGACACTGAGGAATGGACAGGATCAACACTCTTCCACCGGCTCTTCCACCTCCCGATCACTTCCACCATCGCTGGGGGAAGAACGAATGCACCGCGTGG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CACAGCCCCAGATGGTGACTACTGGAGACTGCTGCCCCCAGGTATCCACATTGTCATTGCCCAAGCCCCTGGCTACGCCAAAGTCATCAAGAAAGTCATCATCCCCGCCCGGATGAAGAGGGCTGGCCGTGTGGACTTCATTCTGCAACCTCTGGGGATGGGACCCAAGAACTTTATTCATGGGCTGCGGAGGACTGGGCCCCACGACCCACTGGGAGGTGCCAGCTCTTTGGGGGAGGCCACGGAGCCCGACCCGCTCCGGGCGCGCAGGCAGCCCTCGGCCGACGGGAGTAAGCCCTG",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "AATCGTGTAATTCCCGCTTCCTATGGCAAGCGGCGCCCAGTTCTGGGTGACTTGTGAAATGAGCGTCCTGCTAGGCCATAGTCTCTACGAGGAGGATATAATGTAAAGTCTGGAGCGAACGTAAATGCCGTGACGTGTATTTGTGAGACTTCGCCCGGTCGGTGAGCTGGGTGCATAGGTTCCGAGCGCGCGACCACTTAGCTTCCCTTCTACAACAACCACCTGTGTGCGCATGCCCTCTCATCTTACAGTTGTCAAAGTGCTTTTGCCAGAGCGGTCACATAATCAACCTGAAAGAAG",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TCCTTAGAATATTGGGCATACAATTTACTTTTATAAATGTCTATGAAATAAAATTCTTATATAATTATATAAAATTACATAATATATAATTATATTGAAATAATTATAACTGTTGAATAAATGTATTATTTTTCCTTTGAACTATTTAACAGGGTTATATAACAAACTCATCACGAGTTGTGTCTAATAAGTCATCAGAGTTACTGTTTGACTTGACCCAGGATACAGGATTATCACATTACCAAGGGGGACCAACACTTTCTATGGCAGGTTGGTTTTAGTATTTTTTTCATAAAGGTT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "CGCGCGCGCGCCGGGCCGAGGGGCCGAGAGGCGGGGTCTTACAGCGACCGCGGGAAGAGGGCGCCCCAGAGCTGAGCCGGAGGCCGGCTGCCAGCGGGGCGAGGGTGGGCGCGAGCGCAGGGGCGGGCCCCGAGGGAGGGCGGGGACGGTGAGGGGGCGGGGTCGGGCTAGGCGGGGACGCGCTCGCGGGGTGGGGAAGGCGGGGGCGCGGCGGTGGCGGGAGCGTGCCCGGTCCCCGCCCCTGTTCCCACTCTCCTTCCACCTCGGACCGGCCGGGGCTCCGCAGAGCCAAAGCTCGCT",
"output": "promoter",
"model_response": "promoterpromoterpromo"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "TATTTCCGATTCCTTGACTAACGTACGGGGGCCCAGCACTTTCATTCCATCCCATCACCCCATGATCCTTCCCCAGACTCTAATACTGGCCGTAATATTTACCCCCCCCTGCTCCATGTGCGTAGATCAATGACACGAGCAAGCCATTGGTCTAAAACCCAGGGTTCTCTCGCAGAGGCCCTCCATTGTCTCTGCGGTGTCACTTCGCGCACACACATCAGAGCGGGCCTCTCTGTGAGGCGCGGGGTTGATGTACTCACACGCTGCGTTGATCGGAGGTCGCTGCCAAGCATGGCGCCC",
"output": "Non-promoter",
"model_response": "Non-promoter"
},
{
"instruction": "Determine core promoter detection of following dna sequence, The result will be one of the following: Non-promoter, promoter.",
"input": "GGTCAGCAAAGGCTTCCCTGAGGAATTGGGAATAGCCAGCCATACGGGTGGAGCCGGTCAACTCCTTGTGGTTGGGAGCTAACCGATAGCCCCTACAAGGAGCATTCCAACTATGTTATATAACATTGTGATTTTCGCGAAGGCAACGACTTATTCTTAAGACAGTATCTATCTGAATCTGAAGATCCTTGGTCTGGAGATTGGTAAAAATCTCCAGTCATCTTATGTTTCTATTATACTCTACTGGCGTTTTTACGTGACGCGGGGAGTCTACCTCCCCCAACCATAATTTAACAGTAA",
"output": "Non-promoter",
"model_response": "Non-promoter"
}
]
04-gene-sft/.ipynb_checkpoints/llama_sft_test-checkpoint.ipynb
+1627
View File
File diff suppressed because it is too large Load Diff
04-gene-sft/.ipynb_checkpoints/merge_llama_with_dna_lora-checkpoint.py
+368
View File
@@ -0,0 +1,368 @@
"""
Usage:
python merge_llama_with_chinese_lora.py \
--base_model path/to/llama/model \
--lora_model path/to/first/lora/model [path/to/second/lora/model] \
--output_type [pth|huggingface] \
--output_dir path/to/output/dir
"""
import subprocess
import os
# 设置环境变量, autodl一般区域
result = subprocess.run('bash -c "source /etc/network_turbo && env | grep proxy"', shell=True, capture_output=True, text=True)
output = result.stdout
for line in output.splitlines():
if '=' in line:
var, value = line.split('=', 1)
os.environ[var] = value
import argparse
import json
import os
import gc
import torch
import peft
from peft import PeftModel
from transformers import LlamaForCausalLM, LlamaTokenizer
from huggingface_hub import hf_hub_download
parser = argparse.ArgumentParser()
parser.add_argument('--base_model', default=None, required=True,
type=str, help="Please specify a base_model")
parser.add_argument('--lora_model', default=None, required=True,
type=str, help="Please specify LoRA models to be merged (ordered); use commas to separate multiple LoRA models.")
parser.add_argument('--offload_dir', default=None, type=str,
help="(Optional) Please specify a temp folder for offloading (useful for low-RAM machines). Default None (disable offload).")
parser.add_argument('--output_type', default='pth',choices=['pth','huggingface'], type=str,
help="save the merged model in pth or huggingface format.")
parser.add_argument('--output_dir', default='./', type=str)
emb_to_model_size = {
4096 : '7B',
5120 : '13B',
6656 : '33B',
8192 : '65B',
}
num_shards_of_models = {'7B': 1, '13B': 2, '33B': 4, '65B': 8}
params_of_models = {
'7B':
{
"dim": 4096,
"multiple_of": 256,
"n_heads": 32,
"n_layers": 32,
"norm_eps": 1e-06,
"vocab_size": -1,
},
'13B':
{
"dim": 5120,
"multiple_of": 256,
"n_heads": 40,
"n_layers": 40,
"norm_eps": 1e-06,
"vocab_size": -1,
},
'33B':
{
"dim": 6656,
"multiple_of": 256,
"n_heads": 52,
"n_layers": 60,
"norm_eps": 1e-06,
"vocab_size": -1,
},
'65B':
{
"dim": 8192,
"multiple_of": 256,
"n_heads": 64,
"n_layers": 80,
"norm_eps": 1e-05,
"vocab_size": -1,
},
}
def transpose(weight, fan_in_fan_out):
return weight.T if fan_in_fan_out else weight
# Borrowed and modified from https://github.com/tloen/alpaca-lora
def translate_state_dict_key(k):
k = k.replace("base_model.model.", "")
if k == "model.embed_tokens.weight":
return "tok_embeddings.weight"
elif k == "model.norm.weight":
return "norm.weight"
elif k == "lm_head.weight":
return "output.weight"
elif k.startswith("model.layers."):
layer = k.split(".")[2]
if k.endswith(".self_attn.q_proj.weight"):
return f"layers.{layer}.attention.wq.weight"
elif k.endswith(".self_attn.k_proj.weight"):
return f"layers.{layer}.attention.wk.weight"
elif k.endswith(".self_attn.v_proj.weight"):
return f"layers.{layer}.attention.wv.weight"
elif k.endswith(".self_attn.o_proj.weight"):
return f"layers.{layer}.attention.wo.weight"
elif k.endswith(".mlp.gate_proj.weight"):
return f"layers.{layer}.feed_forward.w1.weight"
elif k.endswith(".mlp.down_proj.weight"):
return f"layers.{layer}.feed_forward.w2.weight"
elif k.endswith(".mlp.up_proj.weight"):
return f"layers.{layer}.feed_forward.w3.weight"
elif k.endswith(".input_layernorm.weight"):
return f"layers.{layer}.attention_norm.weight"
elif k.endswith(".post_attention_layernorm.weight"):
return f"layers.{layer}.ffn_norm.weight"
elif k.endswith("rotary_emb.inv_freq") or "lora" in k:
return None
else:
print(layer, k)
raise NotImplementedError
else:
print(k)
raise NotImplementedError
def unpermute(w):
return (
w.view(n_heads, 2, dim // n_heads // 2, dim).transpose(1, 2).reshape(dim, dim)
)
def save_shards(model_sd, num_shards: int):
# Add the no_grad context manager
with torch.no_grad():
if num_shards == 1:
new_state_dict = {}
for k, v in model_sd.items():
new_k = translate_state_dict_key(k)
if new_k is not None:
if "wq" in new_k or "wk" in new_k:
new_state_dict[new_k] = unpermute(v)
else:
new_state_dict[new_k] = v
os.makedirs(output_dir, exist_ok=True)
print(f"Saving shard 1 of {num_shards} into {output_dir}/consolidated.00.pth")
torch.save(new_state_dict, output_dir + "/consolidated.00.pth")
with open(output_dir + "/params.json", "w") as f:
json.dump(params, f)
else:
new_state_dicts = [dict() for _ in range(num_shards)]
for k in list(model_sd.keys()):
v = model_sd[k]
new_k = translate_state_dict_key(k)
if new_k is not None:
if new_k=='tok_embeddings.weight':
print(f"Processing {new_k}")
assert v.size(1)%num_shards==0
splits = v.split(v.size(1)//num_shards,dim=1)
elif new_k=='output.weight':
print(f"Processing {new_k}")
if v.size(0)%num_shards==0:
splits = v.split(v.size(0)//num_shards,dim=0)
else:
size_list = [v.size(0)//num_shards] * num_shards
size_list[-1] += v.size(0)%num_shards
splits = v.split(size_list, dim=0) # 13B: size_list == [24976,24977]
elif new_k=='norm.weight':
print(f"Processing {new_k}")
splits = [v] * num_shards
elif 'ffn_norm.weight' in new_k:
print(f"Processing {new_k}")
splits = [v] * num_shards
elif 'attention_norm.weight' in new_k:
print(f"Processing {new_k}")
splits = [v] * num_shards
elif 'w1.weight' in new_k:
print(f"Processing {new_k}")
splits = v.split(v.size(0)//num_shards,dim=0)
elif 'w2.weight' in new_k:
print(f"Processing {new_k}")
splits = v.split(v.size(1)//num_shards,dim=1)
elif 'w3.weight' in new_k:
print(f"Processing {new_k}")
splits = v.split(v.size(0)//num_shards,dim=0)
elif 'wo.weight' in new_k:
print(f"Processing {new_k}")
splits = v.split(v.size(1)//num_shards,dim=1)
elif 'wv.weight' in new_k:
print(f"Processing {new_k}")
splits = v.split(v.size(0)//num_shards,dim=0)
elif "wq.weight" in new_k or "wk.weight" in new_k:
print(f"Processing {new_k}")
v = unpermute(v)
splits = v.split(v.size(0)//num_shards,dim=0)
else:
print(f"Unexpected key {new_k}")
raise ValueError
for sd,split in zip(new_state_dicts,splits):
sd[new_k] = split.clone()
del split
del splits
del model_sd[k],v
gc.collect() # Effectively enforce garbage collection
os.makedirs(output_dir, exist_ok=True)
for i,new_state_dict in enumerate(new_state_dicts):
print(f"Saving shard {i+1} of {num_shards} into {output_dir}/consolidated.0{i}.pth")
torch.save(new_state_dict, output_dir + f"/consolidated.0{i}.pth")
with open(output_dir + "/params.json", "w") as f:
print(f"Saving params.json into {output_dir}/params.json")
json.dump(params, f)
if __name__=='__main__':
args = parser.parse_args()
base_model_path = args.base_model
lora_model_paths = [s.strip() for s in args.lora_model.split(',') if len(s.strip())!=0]
output_dir = args.output_dir
output_type = args.output_type
offload_dir = args.offload_dir
print(f"Base model: {base_model_path}")
print(f"LoRA model(s) {lora_model_paths}:")
if offload_dir is not None:
# Load with offloading, which is useful for low-RAM machines.
# Note that if you have enough RAM, please use original method instead, as it is faster.
base_model = LlamaForCausalLM.from_pretrained(
base_model_path,
load_in_8bit=False,
torch_dtype=torch.float16,
offload_folder=offload_dir,
offload_state_dict=True,
low_cpu_mem_usage=True,
device_map={"": "cpu"},
)
else:
# Original method without offloading
base_model = LlamaForCausalLM.from_pretrained(
base_model_path,
load_in_8bit=False,
torch_dtype=torch.float16,
device_map={"": "cpu"},
cache_dir=None, # 不使用缓存目录
force_download=False, # 禁止从远程下载
local_files_only=True # 强制仅从本地文件加载
)
## infer the model size from the checkpoint
embedding_size = base_model.get_input_embeddings().weight.size(1)
model_size = emb_to_model_size[embedding_size]
print(f"Peft version: {peft.__version__}")
print(f"Loading LoRA for {model_size} model")
lora_model = None
lora_model_sd = None
for lora_index, lora_model_path in enumerate(lora_model_paths):
print(f"Loading LoRA {lora_model_path}...")
tokenizer = LlamaTokenizer.from_pretrained(lora_model_path,
cache_dir=None, # 不使用缓存目录
force_download=False, # 禁止从远程下载
local_files_only=True # 强制仅从本地文件加载
)
print(f"base_model vocab size: {base_model.get_input_embeddings().weight.size(0)}")
print(f"tokenizer vocab size: {len(tokenizer)}")
model_vocab_size = base_model.get_input_embeddings().weight.size(0)
assert len(tokenizer) >= model_vocab_size, \
(f"The vocab size of the tokenizer {len(tokenizer)} is smaller than the vocab size of the base model {model_vocab_size}\n"
"This is not the intended use. Please check your model and tokenizer.")
if model_vocab_size != len(tokenizer):
base_model.resize_token_embeddings(len(tokenizer))
print(f"Extended vocabulary size to {len(tokenizer)}")
first_weight = base_model.model.layers[0].self_attn.q_proj.weight
first_weight_old = first_weight.clone()
print(f"Loading LoRA weights")
if hasattr(peft.LoraModel,'merge_and_unload'):
try:
lora_model = PeftModel.from_pretrained(
base_model,
lora_model_path,
device_map={"": "cpu"},
torch_dtype=torch.float16,
local_files_only=True
)
except RuntimeError as e:
if '[49953, 4096]' in str(e):
print("The vocab size of the tokenizer does not match the vocab size of the LoRA weight. \n"
"Did you misuse the LLaMA tokenizer with the Alpaca-LoRA weight?\n"
"Make sure that you use LLaMA tokenizer with the LLaMA-LoRA weight and Alpaca tokenizer with the Alpaca-LoRA weight!")
raise e
assert torch.allclose(first_weight_old, first_weight)
print(f"Merging with merge_and_unload...")
base_model = lora_model.merge_and_unload()
else:
base_model_sd = base_model.state_dict()
try:
lora_model_sd = torch.load(os.path.join(lora_model_path,'adapter_model.bin'),map_location='cpu')
except FileNotFoundError:
print("Cannot find lora model on the disk. Downloading lora model from hub...")
filename = hf_hub_download(repo_id=lora_model_path,filename='adapter_model.bin')
lora_model_sd = torch.load(filename,map_location='cpu')
if 'base_model.model.model.embed_tokens.weight' in lora_model_sd:
assert lora_model_sd['base_model.model.model.embed_tokens.weight'].shape[0]==len(tokenizer), \
("The vocab size of the tokenizer does not match the vocab size of the LoRA weight. \n"
"Did you misuse the LLaMA tokenizer with the Alpaca-LoRA weight?\n"
"Make sure that you use LLaMA tokenizer with the LLaMA-LoRA weight and Alpaca tokenizer with the Alpaca-LoRA weight!")
lora_config = peft.LoraConfig.from_pretrained(lora_model_path)
lora_scaling = lora_config.lora_alpha / lora_config.r
fan_in_fan_out = lora_config.fan_in_fan_out
lora_keys = [k for k in lora_model_sd if 'lora_A' in k]
non_lora_keys = [k for k in lora_model_sd if not 'lora_' in k]
for k in non_lora_keys:
print(f"merging {k}")
original_k = k.replace('base_model.model.','')
base_model_sd[original_k].copy_(lora_model_sd[k])
for k in lora_keys:
print(f"merging {k}")
original_key = k.replace('.lora_A','').replace('base_model.model.','')
assert original_key in base_model_sd
lora_a_key = k
lora_b_key = k.replace('lora_A','lora_B')
base_model_sd[original_key] += (
transpose(lora_model_sd[lora_b_key].float() @ lora_model_sd[lora_a_key].float(),fan_in_fan_out) * lora_scaling
)
assert base_model_sd[original_key].dtype == torch.float16
# did we do anything?
assert not torch.allclose(first_weight_old, first_weight)
tokenizer.save_pretrained(output_dir)
if output_type=='huggingface':
print("Saving to Hugging Face format...")
LlamaForCausalLM.save_pretrained(base_model, output_dir) #, state_dict=deloreanized_sd)
else: # output_type=='pth
print("Saving to pth format...")
base_model_sd = base_model.state_dict()
del lora_model, base_model, lora_model_sd
params = params_of_models[model_size]
num_shards = num_shards_of_models[model_size]
n_layers = params["n_layers"]
n_heads = params["n_heads"]
dim = params["dim"]
dims_per_head = dim // n_heads
base = 10000.0
inv_freq = 1.0 / (base ** (torch.arange(0, dims_per_head, 2).float() / dims_per_head))
save_shards(model_sd=base_model_sd, num_shards=num_shards)
04-gene-sft/.ipynb_checkpoints/merge_pt_model-checkpoint.sh
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
python merge_llama_with_dna_lora.py \
--base_model llama-7b-hf \
--lora_model dnahlm_llama_7b/pt_lora_model \
--output_type huggingface \
--output_dir dnahlm-merge-hf
04-gene-sft/.ipynb_checkpoints/merge_sft_model-checkpoint.sh
Executable
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
python merge_llama_with_dna_lora.py \
--base_model dnahlm-merge-hf \
--lora_model dnahlm-llama7b-sft/sft_lora_model \
--output_type huggingface \
--output_dir dnahlm-llama-7b-sft-v0
04-gene-sft/.ipynb_checkpoints/pip_list-checkpoint.txt
+196
View File
@@ -0,0 +1,196 @@
Package Version
------------------------------ --------------
absl-py 2.1.0
accelerate 1.0.0
aiohappyeyeballs 2.4.4
aiohttp 3.11.11
aiosignal 1.3.2
anaconda-anon-usage 0.4.4
annotated-types 0.7.0
anyio 4.4.0
archspec 0.2.3
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asttokens 2.4.1
async-lru 2.0.4
attrs 23.2.0
Babel 2.15.0
beautifulsoup4 4.12.3
bleach 6.1.0
boltons 23.0.0
Brotli 1.0.9
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 2.0.4
cognite-jupyterlab-copilot 3.5.11
comm 0.2.2
conda 24.4.0
conda-content-trust 0.2.0
conda-libmamba-solver 24.1.0
conda-package-handling 2.2.0
conda_package_streaming 0.9.0
contourpy 1.2.1
cryptography 42.0.5
cycler 0.12.1
datasets 3.2.0
debugpy 1.8.1
decorator 5.1.1
deepspeed 0.15.2
defusedxml 0.7.1
dill 0.3.8
distro 1.9.0
einops 0.8.0
evaluate 0.4.3
executing 2.0.1
fastjsonschema 2.19.1
filelock 3.14.0
fonttools 4.53.0
fqdn 1.5.1
frozenlist 1.5.0
fsspec 2024.5.0
google 3.0.0
grpcio 1.64.0
h11 0.14.0
hjson 3.1.0
httpcore 1.0.5
httpx 0.27.0
huggingface-hub 0.27.0
idna 3.7
ipykernel 6.29.4
ipython 8.25.0
ipywidgets 8.1.3
isoduration 20.11.0
jedi 0.19.1
Jinja2 3.1.4
joblib 1.4.2
json5 0.9.25
jsonpatch 1.33
jsonpointer 2.1
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
jupyter_client 8.6.2
jupyter_core 5.7.2
jupyter-events 0.10.0
jupyter-lsp 2.2.5
jupyter_server 2.14.1
jupyter_server_terminals 0.5.3
jupyterlab 4.2.1
jupyterlab-language-pack-zh-CN 4.2.post1
jupyterlab_pygments 0.3.0
jupyterlab_server 2.27.2
jupyterlab_widgets 3.0.11
kiwisolver 1.4.5
libmambapy 1.5.8
Markdown 3.6
MarkupSafe 2.1.5
matplotlib 3.9.0
matplotlib-inline 0.1.7
menuinst 2.0.2
mistune 3.0.2
mpmath 1.3.0
msgpack 1.1.0
multidict 6.1.0
multiprocess 0.70.16
nbclient 0.10.0
nbconvert 7.16.4
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.3
ninja 1.11.1.3
notebook_shim 0.2.4
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-ml-py 12.560.30
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.5.40
nvidia-nvtx-cu12 12.1.105
overrides 7.7.0
packaging 23.2
pandas 2.2.3
pandocfilters 1.5.1
parso 0.8.4
peft 0.3.0.dev0
pexpect 4.9.0
pillow 10.3.0
pip 24.3.1
platformdirs 3.10.0
pluggy 1.0.0
prometheus_client 0.20.0
prompt_toolkit 3.0.45
propcache 0.2.1
protobuf 5.27.0
psutil 5.9.8
ptyprocess 0.7.0
pure-eval 0.2.2
py-cpuinfo 9.0.0
pyarrow 18.1.0
pycosat 0.6.6
pycparser 2.21
pydantic 2.10.4
pydantic_core 2.27.2
Pygments 2.18.0
pyparsing 3.1.2
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-json-logger 2.0.7
pytz 2024.2
PyYAML 6.0.1
pyzmq 26.0.3
referencing 0.35.1
regex 2024.11.6
requests 2.32.3
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rpds-py 0.18.1
ruamel.yaml 0.17.21
safetensors 0.4.5
scikit-learn 1.6.0
scipy 1.14.1
seaborn 0.13.2
Send2Trash 1.8.3
sentencepiece 0.2.0
setuptools 69.5.1
six 1.16.0
sniffio 1.3.1
soupsieve 2.5
stack-data 0.6.3
supervisor 4.2.5
sympy 1.12.1
tensorboard 2.16.2
tensorboard-data-server 0.7.2
terminado 0.18.1
threadpoolctl 3.5.0
tinycss2 1.3.0
tokenizers 0.20.3
torch 2.3.0+cu121
torchvision 0.18.0+cu121
tornado 6.4
tqdm 4.67.1
traitlets 5.14.3
transformers 4.45.2
truststore 0.8.0
types-python-dateutil 2.9.0.20240316
typing_extensions 4.12.2
tzdata 2024.2
uri-template 1.3.0
urllib3 2.1.0
validation 0.8.3
wcwidth 0.2.13
webcolors 1.13
webencodings 0.5.1
websocket-client 1.8.0
Werkzeug 3.0.3
wheel 0.43.0
widgetsnbextension 4.0.11
xxhash 3.5.0
yarl 1.18.3
zstandard 0.22.0
04-gene-sft/.ipynb_checkpoints/pretrain_gpt2-checkpoint.py
+114
View File
@@ -0,0 +1,114 @@
# import os
# # 设置环境变量
# os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
# # 打印环境变量以确认设置成功
# print(os.environ.get('HF_ENDPOINT'))
import subprocess
import os
# 设置环境变量, autodl一般区域
result = subprocess.run('bash -c "source /etc/network_turbo && env | grep proxy"', shell=True, capture_output=True, text=True)
output = result.stdout
for line in output.splitlines():
if '=' in line:
var, value = line.split('=', 1)
os.environ[var] = value
import math
from transformers import (
GPT2Config,
GPT2LMHeadModel,
GPT2TokenizerFast,
TrainingArguments,
Trainer,
DataCollatorForLanguageModeling,
)
from datasets import Dataset
from datasets import load_dataset
import evaluate
import numpy as np
from transformers import AutoTokenizer,AutoConfig
# 加载 OpenWebText 数据集
dataset = load_dataset("text", data_files="../01-data_env/data/dna_1g.txt")["train"].train_test_split(test_size=0.01, shuffle=True)
# 定义最大输入长度
max_length = 256
# 数据预处理
def preprocess_function(examples):
return tokenizer(examples["text"], truncation=True, max_length=max_length)
# 初始化 GPT-2 分词器
tokenizer = AutoTokenizer.from_pretrained("gpt2_tokenizer")
tokenized_dataset = dataset.map(preprocess_function, batched=True, num_proc=5)
# 4. 创建一个数据收集器，用于动态填充和遮蔽
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer, mlm=False
)
# 加载并调整 GPT-2 配置
config = AutoConfig.from_pretrained(
"gpt2", # 加载 GPT-2 的默认配置
vocab_size=len(tokenizer), # 更新词汇表大小为自定义分词器的词汇表大小
n_ctx=max_length, # 最大上下文长度（序列长度）
n_positions=max_length, # 最大位置编码长度，通常与 n_ctx 一致
)
# 初始化 GPT-2 模型
model = GPT2LMHeadModel(config)
# 定义训练参数
training_args = TrainingArguments(
output_dir="./gpt2-small",
overwrite_output_dir=True,
num_train_epochs=5,
per_device_train_batch_size=64,
save_steps=10000,
save_total_limit=2,
logging_dir="./logs",
logging_steps=20000,
evaluation_strategy="steps",
eval_steps=10000,
learning_rate=5e-5,
warmup_steps=500,
weight_decay=0.01,
fp16=True, # 启用混合精度训练
#deepspeed="ds_zero2_no_offload.json"
)
# 初始化 Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset["train"],
eval_dataset=tokenized_dataset["test"],
tokenizer=tokenizer,
data_collator=data_collator,
)
# 开始训练
trainer.train()
# 评估 perplexity
eval_results = trainer.evaluate()
perplexity = math.exp(eval_results["eval_loss"])
print(f"Perplexity: {perplexity}")
out_model_path = "gpt2-small-gene-openweb"
trainer.save_model(out_model_path)
tokenizer.save_pretrained(out_model_path)
04-gene-sft/.ipynb_checkpoints/run_clm_pt_with_peft-checkpoint.py
+648
View File
@@ -0,0 +1,648 @@
#!/usr/bin/env python
# coding=utf-8
# Copyright 2020 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.
Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=text-generation
"""
# You can also adapt this script on your own causal language modeling task. Pointers for this are left as comments.
import subprocess
import os
# 设置环境变量, autodl一般区域
result = subprocess.run('bash -c "source /etc/network_turbo && env | grep proxy"', shell=True, capture_output=True, text=True)
output = result.stdout
for line in output.splitlines():
if '=' in line:
var, value = line.split('=', 1)
os.environ[var] = value
import logging
import numpy as np
import math
import os
import sys
from dataclasses import dataclass, field
from itertools import chain
from typing import Optional, List, Dict, Any, Mapping
from pathlib import Path
import datasets
import torch
from datasets import load_dataset, concatenate_datasets
import transformers
from transformers import (
CONFIG_MAPPING,
MODEL_FOR_CAUSAL_LM_MAPPING,
AutoConfig,
AutoModelForCausalLM,
LlamaForCausalLM,
LlamaTokenizer,
AutoTokenizer,
HfArgumentParser,
Trainer,
TrainingArguments,
is_torch_tpu_available,
set_seed,
)
from transformers.testing_utils import CaptureLogger
from transformers.trainer_utils import get_last_checkpoint
from transformers.utils import send_example_telemetry
from transformers.utils.versions import require_version
from sklearn.metrics import accuracy_score
from peft import LoraConfig, TaskType, get_peft_model, PeftModel, get_peft_model_state_dict
from transformers.trainer_utils import PREFIX_CHECKPOINT_DIR
class SavePeftModelCallback(transformers.TrainerCallback):
def save_model(self, args, state, kwargs):
if state.best_model_checkpoint is not None:
checkpoint_folder = os.path.join(state.best_model_checkpoint, "pt_lora_model")
else:
checkpoint_folder = os.path.join(args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{state.global_step}")
peft_model_path = os.path.join(checkpoint_folder, "pt_lora_model")
kwargs["model"].save_pretrained(peft_model_path)
if "tokenizer" in kwargs:
kwargs["tokenizer"].save_pretrained(peft_model_path)
else:
kwargs["processing_class"].save_pretrained(peft_model_path)
def on_save(self, args, state, control, **kwargs):
self.save_model(args, state, kwargs)
return control
def on_train_end(self, args, state, control, **kwargs):
peft_model_path = os.path.join(args.output_dir, "pt_lora_model")
kwargs["model"].save_pretrained(peft_model_path)
if "tokenizer" in kwargs:
kwargs["tokenizer"].save_pretrained(peft_model_path)
else:
kwargs["processing_class"].save_pretrained(peft_model_path)
def accuracy(predictions, references, normalize=True, sample_weight=None):
return {
"accuracy": float(
accuracy_score(references, predictions, normalize=normalize, sample_weight=sample_weight)
)
}
def compute_metrics(eval_preds):
preds, labels = eval_preds
# preds have the same shape as the labels, after the argmax(-1) has been calculated
# by preprocess_logits_for_metrics but we need to shift the labels
labels = labels[:, 1:].reshape(-1)
preds = preds[:, :-1].reshape(-1)
return accuracy(predictions=preds, references=labels)
def preprocess_logits_for_metrics(logits, labels):
if isinstance(logits, tuple):
# Depending on the model and config, logits may contain extra tensors,
# like past_key_values, but logits always come first
logits = logits[0]
return logits.argmax(dim=-1)
def fault_tolerance_data_collator(features: List) -> Dict[str, Any]:
if not isinstance(features[0], Mapping):
features = [vars(f) for f in features]
first = features[0]
batch = {}
# Special handling for labels.
# Ensure that tensor is created with the correct type
# (it should be automatically the case, but let's make sure of it.)
if "label" in first and first["label"] is not None:
label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
dtype = torch.long if isinstance(label, int) else torch.float
batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
elif "label_ids" in first and first["label_ids"] is not None:
if isinstance(first["label_ids"], torch.Tensor):
batch["labels"] = torch.stack([f["label_ids"] for f in features])
else:
dtype = torch.long if isinstance(first["label_ids"][0], int) else torch.float
batch["labels"] = torch.tensor([f["label_ids"] for f in features], dtype=dtype)
# Handling of all other possible keys.
# Again, we will use the first element to figure out which key/values are not None for this model.
try:
for k, v in first.items():
if k not in ("label", "label_ids") and v is not None and not isinstance(v, str):
if isinstance(v, torch.Tensor):
batch[k] = torch.stack([f[k] for f in features])
elif isinstance(v, np.ndarray):
batch[k] = torch.tensor(np.stack([f[k] for f in features]))
else:
batch[k] = torch.tensor([f[k] for f in features])
except ValueError: # quick fix by simply take the first example
for k, v in first.items():
if k not in ("label", "label_ids") and v is not None and not isinstance(v, str):
if isinstance(v, torch.Tensor):
batch[k] = torch.stack([features[0][k]] * len(features))
elif isinstance(v, np.ndarray):
batch[k] = torch.tensor(np.stack([features[0][k]] * len(features)))
else:
batch[k] = torch.tensor([features[0][k]] * len(features))
return batch
MODEL_CONFIG_CLASSES = list(MODEL_FOR_CAUSAL_LM_MAPPING.keys())
MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES)
@dataclass
class ModelArguments:
"""
Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.
"""
model_name_or_path: Optional[str] = field(
default=None,
metadata={
"help": (
"The model checkpoint for weights initialization.Don't set if you want to train a model from scratch."
)
},
)
tokenizer_name_or_path: Optional[str] = field(
default=None,
metadata={
"help": (
"The tokenizer for weights initialization.Don't set if you want to train a model from scratch."
)
},
)
model_type: Optional[str] = field(
default=None,
metadata={"help": "If training from scratch, pass a model type from the list: " + ", ".join(MODEL_TYPES)},
)
config_overrides: Optional[str] = field(
default=None,
metadata={
"help": (
"Override some existing default config settings when a model is trained from scratch. Example: "
"n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index"
)
},
)
config_name: Optional[str] = field(
default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
)
tokenizer_name: Optional[str] = field(
default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
)
cache_dir: Optional[str] = field(
default=None,
metadata={"help": "Where do you want to store the pretrained models downloaded from huggingface.co"},
)
use_fast_tokenizer: bool = field(
default=True,
metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
)
model_revision: str = field(
default="main",
metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
)
use_auth_token: bool = field(
default=False,
metadata={
"help": (
"Will use the token generated when running `huggingface-cli login` (necessary to use this script "
"with private models)."
)
},
)
torch_dtype: Optional[str] = field(
default=None,
metadata={
"help": (
"Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the "
"dtype will be automatically derived from the model's weights."
),
"choices": ["auto", "bfloat16", "float16", "float32"],
},
)
def __post_init__(self):
if self.config_overrides is not None and (self.config_name is not None or self.model_name_or_path is not None):
raise ValueError(
"--config_overrides can't be used in combination with --config_name or --model_name_or_path"
)
@dataclass
class DataTrainingArguments:
"""
Arguments pertaining to what data we are going to input our model for training and eval.
"""
dataset_dir: Optional[str] = field(
default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."}
)
dataset_config_name: Optional[str] = field(
default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."}
)
train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."})
validation_file: Optional[str] = field(
default=None,
metadata={"help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."},
)
max_train_samples: Optional[int] = field(
default=None,
metadata={
"help": (
"For debugging purposes or quicker training, truncate the number of training examples to this "
"value if set."
)
},
)
max_eval_samples: Optional[int] = field(
default=None,
metadata={
"help": (
"For debugging purposes or quicker training, truncate the number of evaluation examples to this "
"value if set."
)
},
)
streaming: bool = field(default=False, metadata={"help": "Enable streaming mode"})
block_size: Optional[int] = field(
default=None,
metadata={
"help": (
"Optional input sequence length after tokenization. "
"The training dataset will be truncated in block of this size for training. "
"Default to the model max input length for single sentence inputs (take into account special tokens)."
)
},
)
overwrite_cache: bool = field(
default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
)
validation_split_percentage: Optional[float] = field(
default=0.05,
metadata={
"help": "The percentage of the train set used as validation set in case there's no validation split"
},
)
preprocessing_num_workers: Optional[int] = field(
default=None,
metadata={"help": "The number of processes to use for the preprocessing."},
)
keep_linebreaks: bool = field(
default=True, metadata={"help": "Whether to keep line breaks when using TXT files or not."}
)
data_cache_dir: Optional[str] = field(default="./", metadata={"help": "The datasets processed stored"})
def __post_init__(self):
if self.streaming:
require_version("datasets>=2.0.0", "The streaming feature requires `datasets>=2.0.0`")
@dataclass
class MyTrainingArguments(TrainingArguments):
trainable : Optional[str] = field(default="q_proj,v_proj")
lora_rank : Optional[int] = field(default=8)
lora_dropout : Optional[float] = field(default=0.1)
lora_alpha : Optional[float] = field(default=32.)
modules_to_save : Optional[str] = field(default=None)
debug_mode : Optional[bool] = field(default=False)
peft_path : Optional[str] = field(default=None)
logger = logging.getLogger(__name__)
def main():
parser = HfArgumentParser((ModelArguments, DataTrainingArguments, MyTrainingArguments))
if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
# If we pass only one argument to the script and it's the path to a json file,
# let's parse it to get our arguments.
model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
else:
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
# Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
# information sent is the one passed as arguments along with your Python/PyTorch versions.
send_example_telemetry("run_clm", model_args, data_args)
# Setup logging
logging.basicConfig(format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",datefmt="%m/%d/%Y %H:%M:%S",
level=logging.INFO, # if training_args.local_rank in [-1, 0] else logging.WARN,
handlers=[logging.StreamHandler(sys.stdout)],)
if training_args.should_log:
# The default of training_args.log_level is passive, so we set log level at info here to have that default.
transformers.utils.logging.set_verbosity_info()
log_level = training_args.get_process_log_level()
logger.setLevel(log_level)
datasets.utils.logging.set_verbosity(log_level)
transformers.utils.logging.set_verbosity(log_level)
transformers.utils.logging.enable_default_handler()
transformers.utils.logging.enable_explicit_format()
# transformers.tokenization_utils.logging.set_verbosity_warning()
# Log on each process the small summary:
logger.warning(
f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
+ f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
)
# Detecting last checkpoint.
last_checkpoint = None
if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
last_checkpoint = get_last_checkpoint(training_args.output_dir)
if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
raise ValueError(
f"Output directory ({training_args.output_dir}) already exists and is not empty. "
"Use --overwrite_output_dir to overcome."
)
elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
logger.info(
f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
"the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
)
# Set seed before initializing model.
set_seed(training_args.seed)
config_kwargs = {
"cache_dir": model_args.cache_dir,
"revision": model_args.model_revision,
"use_auth_token": True if model_args.use_auth_token else None,
}
if model_args.config_name:
config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)
elif model_args.model_name_or_path:
config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)
else:
config = CONFIG_MAPPING[model_args.model_type]()
logger.warning("You are instantiating a new config instance from scratch.")
if model_args.config_overrides is not None:
logger.info(f"Overriding config: {model_args.config_overrides}")
config.update_from_string(model_args.config_overrides)
logger.info(f"New config: {config}")
tokenizer_kwargs = {
"cache_dir": model_args.cache_dir,
"use_fast": model_args.use_fast_tokenizer,
"revision": model_args.model_revision,
"use_auth_token": True if model_args.use_auth_token else None,
}
if model_args.tokenizer_name:
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)
elif model_args.tokenizer_name_or_path:
tokenizer = LlamaTokenizer.from_pretrained(model_args.tokenizer_name_or_path, **tokenizer_kwargs)
else:
raise ValueError(
"You are instantiating a new tokenizer from scratch. This is not supported by this script."
"You can do it from another script, save it, and load it from here, using --tokenizer_name."
)
# Preprocessing the datasets.
# First we tokenize all the texts.
# since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function
tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base")
def tokenize_function(examples):
with CaptureLogger(tok_logger) as cl:
output = tokenizer(examples["text"])
# clm input could be much much longer than block_size
if "Token indices sequence length is longer than the" in cl.out:
tok_logger.warning(
"^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits"
" before being passed to the model."
)
return output
if data_args.block_size is None:
block_size = tokenizer.model_max_length
if block_size > 1024:
logger.warning(
"The chosen tokenizer supports a `model_max_length` that is longer than the default `block_size` value"
" of 1024. If you would like to use a longer `block_size` up to `tokenizer.model_max_length` you can"
" override this default with `--block_size xxx`."
)
block_size = 1024
else:
if data_args.block_size > tokenizer.model_max_length:
logger.warning(
f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model"
f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}."
)
block_size = min(data_args.block_size, tokenizer.model_max_length)
# Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.
def group_texts(examples):
# Concatenate all texts.
concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
total_length = len(concatenated_examples[list(examples.keys())[0]])
# We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
# customize this part to your needs.
if total_length >= block_size:
total_length = (total_length // block_size) * block_size
# Split by chunks of max_len.
result = {
k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
for k, t in concatenated_examples.items()
}
result["labels"] = result["input_ids"].copy()
return result
with training_args.main_process_first(desc="dataset map tokenization and grouping"):
lm_datasets = []
path = Path(data_args.dataset_dir)
files = [file.name for file in path.glob("*.txt")]
if training_args.debug_mode is True:
files = [files[0]]
for idx, file in enumerate(files):
data_file = os.path.join(path, file)
filename = ''.join(file.split(".")[:-1])
cache_path = os.path.join(data_args.data_cache_dir, filename)
os.makedirs(cache_path, exist_ok=True)
try:
processed_dataset = datasets.load_from_disk(cache_path, keep_in_memory=False)
logger.info(f'training datasets-{filename} has been loaded from disk')
except Exception:
cache_dir = os.path.join(data_args.data_cache_dir, filename+"_text")
os.makedirs(cache_dir, exist_ok=True)
raw_dataset = load_dataset("text", data_files=data_file, cache_dir=cache_dir, keep_in_memory=False)
logger.info(f"{file} has been loaded")
tokenized_dataset = raw_dataset.map(
tokenize_function,
batched=True,
num_proc=data_args.preprocessing_num_workers,
remove_columns="text",
load_from_cache_file=True,
keep_in_memory=False,
cache_file_names = {k: os.path.join(cache_dir, 'tokenized.arrow') for k in raw_dataset},
desc="Running tokenizer on dataset",
)
grouped_datasets = tokenized_dataset.map(
group_texts,
batched=True,
num_proc=data_args.preprocessing_num_workers,
load_from_cache_file=True,
keep_in_memory=False,
cache_file_names = {k: os.path.join(cache_dir, 'grouped.arrow') for k in tokenized_dataset},
desc=f"Grouping texts in chunks of {block_size}",
)
processed_dataset = grouped_datasets
processed_dataset.save_to_disk(cache_path)
if idx == 0:
lm_datasets = processed_dataset['train']
else:
assert lm_datasets.features.type == processed_dataset["train"].features.type
lm_datasets = concatenate_datasets([lm_datasets, processed_dataset["train"]])
lm_datasets = lm_datasets.train_test_split(test_size = data_args.validation_split_percentage)
if training_args.do_train:
train_dataset = lm_datasets['train']
if data_args.max_train_samples is not None:
max_train_samples = min(len(train_dataset), data_args.max_train_samples)
train_dataset = train_dataset.select(range(max_train_samples))
logger.info(f"Num train_samples {len(train_dataset)}")
logger.info("training example:")
logger.info(tokenizer.decode(train_dataset[0]['input_ids']))
if training_args.do_eval:
eval_dataset = lm_datasets["test"]
if data_args.max_eval_samples is not None:
max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
eval_dataset = eval_dataset.select(range(max_eval_samples))
logger.info(f"Num eval_samples {len(eval_dataset)}")
logger.info("training example:")
logger.info(tokenizer.decode(eval_dataset[0]['input_ids']))
if model_args.model_name_or_path:
torch_dtype = (
model_args.torch_dtype
if model_args.torch_dtype in ["auto", None]
else getattr(torch, model_args.torch_dtype)
)
model = LlamaForCausalLM.from_pretrained(
model_args.model_name_or_path,
from_tf=bool(".ckpt" in model_args.model_name_or_path),
config=config,
cache_dir=model_args.cache_dir,
revision=model_args.model_revision,
use_auth_token=True if model_args.use_auth_token else None,
torch_dtype=torch_dtype,
low_cpu_mem_usage=True
)
else:
model = AutoModelForCausalLM.from_config(config)
n_params = sum({p.data_ptr(): p.numel() for p in model.parameters()}.values())
logger.info(f"Training new model from scratch - Total size={n_params/2**20:.2f}M params")
model_vocab_size = model.get_output_embeddings().weight.size(0)
model.resize_token_embeddings(len(tokenizer))
if training_args.peft_path is not None:
logger.info("Peft from pre-trained model")
model = PeftModel.from_pretrained(model, training_args.peft_path)
else:
logger.info("Init new peft model")
target_modules = training_args.trainable.split(',')
modules_to_save = training_args.modules_to_save
if modules_to_save is not None:
modules_to_save = modules_to_save.split(',')
lora_rank = training_args.lora_rank
lora_dropout = training_args.lora_dropout
lora_alpha = training_args.lora_alpha
logger.info(f"target_modules: {target_modules}")
logger.info(f"lora_rank: {lora_rank}")
peft_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
target_modules=target_modules,
inference_mode=False,
r=lora_rank, lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
modules_to_save=modules_to_save)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
old_state_dict = model.state_dict
model.state_dict = (
lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
).__get__(model, type(model))
# Initialize our Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset if training_args.do_train else None,
eval_dataset=eval_dataset if training_args.do_eval else None,
tokenizer=tokenizer,
data_collator=fault_tolerance_data_collator,
compute_metrics=compute_metrics if training_args.do_eval and not is_torch_tpu_available() else None,
preprocess_logits_for_metrics=preprocess_logits_for_metrics
if training_args.do_eval and not is_torch_tpu_available()
else None,
)
trainer.add_callback(SavePeftModelCallback)
# Training
if training_args.do_train:
checkpoint = None
if training_args.resume_from_checkpoint is not None:
checkpoint = training_args.resume_from_checkpoint
elif last_checkpoint is not None:
checkpoint = last_checkpoint
train_result = trainer.train(resume_from_checkpoint=checkpoint)
metrics = train_result.metrics
max_train_samples = (
data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
)
metrics["train_samples"] = min(max_train_samples, len(train_dataset))
trainer.log_metrics("train", metrics)
trainer.save_metrics("train", metrics)
trainer.save_state()
# Evaluation
if training_args.do_eval:
logger.info("*** Evaluate ***")
metrics = trainer.evaluate()
max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
try:
perplexity = math.exp(metrics["eval_loss"])
except OverflowError:
perplexity = float("inf")
metrics["perplexity"] = perplexity
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
if __name__ == "__main__":
main()
04-gene-sft/.ipynb_checkpoints/run_clm_sft_with_peft-checkpoint.py
+449
View File
@@ -0,0 +1,449 @@
#!/usr/bin/env python
# coding=utf-8
# Copyright 2020 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.
Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=text-generation
"""
# You can also adapt this script on your own causal language modeling task. Pointers for this are left as comments.
import logging
import math
import os
import sys
from dataclasses import dataclass, field
from typing import Optional
from pathlib import Path
import datasets
import torch
from build_dataset import build_instruction_dataset, DataCollatorForSupervisedDataset
import transformers
from transformers import (
CONFIG_MAPPING,
AutoConfig,
AutoModelForCausalLM,
LlamaForCausalLM,
LlamaTokenizer,
AutoTokenizer,
HfArgumentParser,
Trainer,
TrainingArguments,
set_seed,
)
from transformers.trainer_utils import get_last_checkpoint
from transformers.utils import send_example_telemetry
from transformers.utils.versions import require_version
from peft import LoraConfig, TaskType, get_peft_model, PeftModel, get_peft_model_state_dict
from transformers.trainer_utils import PREFIX_CHECKPOINT_DIR
IGNORE_INDEX = -100
DEFAULT_PAD_TOKEN = "[PAD]"
DEFAULT_EOS_TOKEN = "</s>"
DEFAULT_BOS_TOKEN = "<s>"
DEFAULT_UNK_TOKEN = "<unk>"
require_version("datasets>=1.8.0", "To fix: pip install -r examples/pytorch/language-modeling/requirements.txt")
class SavePeftModelCallback(transformers.TrainerCallback):
def save_model(self, args, state, kwargs):
if state.best_model_checkpoint is not None:
checkpoint_folder = os.path.join(state.best_model_checkpoint, "sft_lora_model")
else:
checkpoint_folder = os.path.join(args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{state.global_step}")
peft_model_path = os.path.join(checkpoint_folder, "sft_lora_model")
kwargs["model"].save_pretrained(peft_model_path)
if "tokenizer" in kwargs:
kwargs["tokenizer"].save_pretrained(peft_model_path)
else:
kwargs["processing_class"].save_pretrained(peft_model_path)
def on_save(self, args, state, control, **kwargs):
self.save_model(args, state, kwargs)
return control
def on_train_end(self, args, state, control, **kwargs):
peft_model_path = os.path.join(args.output_dir, "sft_lora_model")
kwargs["model"].save_pretrained(peft_model_path)
if "tokenizer" in kwargs:
kwargs["tokenizer"].save_pretrained(peft_model_path)
else:
kwargs["processing_class"].save_pretrained(peft_model_path)
@dataclass
class ModelArguments:
"""
Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.
"""
model_name_or_path: Optional[str] = field(
default=None,
metadata={
"help": (
"The model checkpoint for weights initialization.Don't set if you want to train a model from scratch."
)
},
)
tokenizer_name_or_path: Optional[str] = field(
default=None,
metadata={
"help": (
"The tokenizer for weights initialization.Don't set if you want to train a model from scratch."
)
},
)
config_overrides: Optional[str] = field(
default=None,
metadata={
"help": (
"Override some existing default config settings when a model is trained from scratch. Example: "
"n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index"
)
},
)
config_name: Optional[str] = field(
default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
)
tokenizer_name: Optional[str] = field(
default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
)
cache_dir: Optional[str] = field(
default=None,
metadata={"help": "Where do you want to store the pretrained models downloaded from huggingface.co"},
)
use_fast_tokenizer: bool = field(
default=True,
metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
)
model_revision: str = field(
default="main",
metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
)
use_auth_token: bool = field(
default=False,
metadata={
"help": (
"Will use the token generated when running `huggingface-cli login` (necessary to use this script "
"with private models)."
)
},
)
torch_dtype: Optional[str] = field(
default=None,
metadata={
"help": (
"Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the "
"dtype will be automatically derived from the model's weights."
),
"choices": ["auto", "bfloat16", "float16", "float32"],
},
)
def __post_init__(self):
if self.config_overrides is not None and (self.config_name is not None or self.model_name_or_path is not None):
raise ValueError(
"--config_overrides can't be used in combination with --config_name or --model_name_or_path"
)
@dataclass
class DataTrainingArguments:
"""
Arguments pertaining to what data we are going to input our model for training and eval.
"""
dataset_dir: Optional[str] = field(
default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."}
)
train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."})
validation_file: Optional[str] = field(
default=None,
metadata={"help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."},
)
overwrite_cache: bool = field(
default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
)
validation_split_percentage: Optional[float] = field(
default=0.05,
metadata={
"help": "The percentage of the train set used as validation set in case there's no validation split"
},
)
preprocessing_num_workers: Optional[int] = field(
default=None,
metadata={"help": "The number of processes to use for the preprocessing."},
)
keep_linebreaks: bool = field(
default=True, metadata={"help": "Whether to keep line breaks when using TXT files or not."}
)
data_cache_dir: Optional[str] = field(default=None, metadata={"help": "The datasets processed stored"})
max_seq_length: Optional[int] = field(default=512)
@dataclass
class MyTrainingArguments(TrainingArguments):
trainable : Optional[str] = field(default="q_proj,v_proj")
lora_rank : Optional[int] = field(default=8)
lora_dropout : Optional[float] = field(default=0.1)
lora_alpha : Optional[float] = field(default=32.)
modules_to_save : Optional[str] = field(default=None)
peft_path : Optional[str] = field(default=None)
force_resize_embeddings: bool = field(default=False)
logger = logging.getLogger(__name__)
def main():
parser = HfArgumentParser((ModelArguments, DataTrainingArguments, MyTrainingArguments))
if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
# If we pass only one argument to the script and it's the path to a json file,
# let's parse it to get our arguments.
model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
else:
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
send_example_telemetry("run_clm", model_args, data_args)
# Setup logging
logging.basicConfig(format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",datefmt="%m/%d/%Y %H:%M:%S",
level=logging.INFO, # if training_args.local_rank in [-1, 0] else logging.WARN,
handlers=[logging.StreamHandler(sys.stdout)],)
if training_args.should_log:
# The default of training_args.log_level is passive, so we set log level at info here to have that default.
transformers.utils.logging.set_verbosity_info()
log_level = training_args.get_process_log_level()
logger.setLevel(log_level)
datasets.utils.logging.set_verbosity(log_level)
transformers.utils.logging.set_verbosity(log_level)
transformers.utils.logging.enable_default_handler()
transformers.utils.logging.enable_explicit_format()
# transformers.tokenization_utils.logging.set_verbosity_warning()
# Log on each process the small summary:
logger.warning(
f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
+ f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
)
# Detecting last checkpoint.
last_checkpoint = None
if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
last_checkpoint = get_last_checkpoint(training_args.output_dir)
if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
raise ValueError(
f"Output directory ({training_args.output_dir}) already exists and is not empty. "
"Use --overwrite_output_dir to overcome."
)
elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
logger.info(
f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
"the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
)
# Set seed before initializing model.
set_seed(training_args.seed)
config_kwargs = {
"cache_dir": model_args.cache_dir,
"revision": model_args.model_revision,
"use_auth_token": True if model_args.use_auth_token else None,
}
if model_args.config_name:
config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)
elif model_args.model_name_or_path:
config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)
else:
config = CONFIG_MAPPING[model_args.model_type]()
logger.warning("You are instantiating a new config instance from scratch.")
if model_args.config_overrides is not None:
logger.info(f"Overriding config: {model_args.config_overrides}")
config.update_from_string(model_args.config_overrides)
logger.info(f"New config: {config}")
tokenizer_kwargs = {
"cache_dir": model_args.cache_dir,
"use_fast": model_args.use_fast_tokenizer,
"revision": model_args.model_revision,
"use_auth_token": True if model_args.use_auth_token else None,
}
if model_args.tokenizer_name:
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)
elif model_args.tokenizer_name_or_path:
tokenizer = LlamaTokenizer.from_pretrained(model_args.tokenizer_name_or_path, **tokenizer_kwargs)
else:
raise ValueError(
"You are instantiating a new tokenizer from scratch. This is not supported by this script."
"You can do it from another script, save it, and load it from here, using --tokenizer_name."
)
if tokenizer.pad_token is None:
print(f"Adding pad token {DEFAULT_PAD_TOKEN}")
tokenizer.add_special_tokens(dict(pad_token=DEFAULT_PAD_TOKEN))
data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
eval_dataset=None
train_dataset = None
if training_args.do_train:
with training_args.main_process_first(desc="loading and tokenization"):
path = Path(data_args.dataset_dir)
files = [os.path.join(path,file.name) for file in path.glob("*.json")]
logger.info(f"Training files: {' '.join(files)}")
train_dataset = build_instruction_dataset(
data_path=files,
tokenizer=tokenizer,
max_seq_length=data_args.max_seq_length,
data_cache_dir = None,
preprocessing_num_workers = data_args.preprocessing_num_workers)
logger.info(f"Num train_samples {len(train_dataset)}")
logger.info("training example:")
logger.info(tokenizer.decode(train_dataset[0]['input_ids']))
if training_args.do_eval:
with training_args.main_process_first(desc="loading and tokenization"):
files = [data_args.validation_file]
logger.info(f"Evaluation files: {' '.join(files)}")
eval_dataset = build_instruction_dataset(
data_path=files,
tokenizer=tokenizer,
max_seq_length=data_args.max_seq_length,
data_cache_dir = None,
preprocessing_num_workers = data_args.preprocessing_num_workers)
logger.info(f"Num eval_samples {len(eval_dataset)}")
logger.info("eval example:")
logger.info(tokenizer.decode(eval_dataset[0]['input_ids']))
if model_args.model_name_or_path:
torch_dtype = (
model_args.torch_dtype
if model_args.torch_dtype in ["auto", None]
else getattr(torch, model_args.torch_dtype)
)
model = LlamaForCausalLM.from_pretrained(
model_args.model_name_or_path,
from_tf=bool(".ckpt" in model_args.model_name_or_path),
config=config,
cache_dir=model_args.cache_dir,
revision=model_args.model_revision,
use_auth_token=True if model_args.use_auth_token else None,
torch_dtype=torch_dtype,
low_cpu_mem_usage=True
)
else:
model = AutoModelForCausalLM.from_config(config)
n_params = sum({p.data_ptr(): p.numel() for p in model.parameters()}.values())
logger.info(f"Training new model from scratch - Total size={n_params/2**20:.2f}M params")
logger.info(f"len(tokenizer):{len(tokenizer)}")
embedding_size = model.get_input_embeddings().weight.shape[0]
if len(tokenizer) != embedding_size:
logger.info("resize the embedding size by the size of the tokenizer")
model.resize_token_embeddings(len(tokenizer))
if training_args.peft_path is not None:
logger.info("Peft from pre-trained model")
model = PeftModel.from_pretrained(model, training_args.peft_path)
else:
logger.info("Init new peft model")
target_modules = training_args.trainable.split(',')
modules_to_save = training_args.modules_to_save
if modules_to_save is not None:
modules_to_save = modules_to_save.split(',')
lora_rank = training_args.lora_rank
lora_dropout = training_args.lora_dropout
lora_alpha = training_args.lora_alpha
logger.info(f"target_modules: {target_modules}")
logger.info(f"lora_rank: {lora_rank}")
peft_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
target_modules=target_modules,
inference_mode=False,
r=lora_rank, lora_alpha=lora_alpha,
lora_dropout=lora_dropout,
modules_to_save=modules_to_save)
model = get_peft_model(model, peft_config)
#model.base_model.tie_weights()
model.print_trainable_parameters()
logger.info(f"model.modules_to_save: {model.modules_to_save}")
old_state_dict = model.state_dict
model.state_dict = (
lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
).__get__(model, type(model))
# Initialize our Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
data_collator=data_collator,
)
trainer.add_callback(SavePeftModelCallback)
# Training
if training_args.do_train:
checkpoint = None
if training_args.resume_from_checkpoint is not None:
checkpoint = training_args.resume_from_checkpoint
elif last_checkpoint is not None:
checkpoint = last_checkpoint
train_result = trainer.train(resume_from_checkpoint=checkpoint)
metrics = train_result.metrics
metrics["train_samples"] = len(train_dataset)
trainer.log_metrics("train", metrics)
trainer.save_metrics("train", metrics)
trainer.save_state()
# Evaluation
if training_args.do_eval:
logger.info("*** Evaluate ***")
metrics = trainer.evaluate()
metrics["eval_samples"] =len(eval_dataset)
try:
perplexity = math.exp(metrics["eval_loss"])
except OverflowError:
perplexity = float("inf")
metrics["perplexity"] = perplexity
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)
if __name__ == "__main__":
main()
04-gene-sft/.ipynb_checkpoints/run_pt-checkpoint.sh
Executable
+55
View File
@@ -0,0 +1,55 @@
lr=2e-4
lora_rank=8
lora_alpha=32
lora_trainable="q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj"
modules_to_save="embed_tokens,lm_head"
lora_dropout=0.05
pretrained_model=./llama-7b-hf
dna_eng_tokenizer_path=./merged_gene_eng_tokenizer_hf
dataset_dir=./train_data
data_cache=temp_data_cache_dir
per_device_train_batch_size=4
per_device_eval_batch_size=4
gradient_accumulation_steps=8
output_dir=dnahlm_llama_7b
deepspeed_config_file=ds_zero2_no_offload.json
torchrun --nnodes 1 --nproc_per_node 10 run_clm_pt_with_peft.py \
--deepspeed ${deepspeed_config_file} \
--model_name_or_path ${pretrained_model} \
--tokenizer_name_or_path ${dna_eng_tokenizer_path} \
--dataset_dir ${dataset_dir} \
--data_cache_dir ${data_cache} \
--validation_split_percentage 0.001 \
--per_device_train_batch_size ${per_device_train_batch_size} \
--per_device_eval_batch_size ${per_device_eval_batch_size} \
--do_train \
--seed $RANDOM \
--fp16 \
--num_train_epochs 1 \
--lr_scheduler_type cosine \
--learning_rate ${lr} \
--warmup_ratio 0.05 \
--weight_decay 0.01 \
--logging_strategy steps \
--logging_steps 10 \
--save_strategy steps \
--save_total_limit 3 \
--save_steps 200 \
--gradient_accumulation_steps ${gradient_accumulation_steps} \
--preprocessing_num_workers 64 \
--block_size 256 \
--output_dir ${output_dir} \
--overwrite_output_dir \
--ddp_timeout 30000 \
--logging_first_step True \
--lora_rank ${lora_rank} \
--lora_alpha ${lora_alpha} \
--trainable ${lora_trainable} \
--modules_to_save ${modules_to_save} \
--lora_dropout ${lora_dropout} \
--torch_dtype float16 \
--gradient_checkpointing \
--ddp_find_unused_parameters False
04-gene-sft/.ipynb_checkpoints/run_sft-checkpoint.sh
Executable
+59
View File
@@ -0,0 +1,59 @@
lr=1e-4
lora_rank=8
lora_alpha=32
lora_trainable="q_proj,v_proj,k_proj,o_proj,gate_proj,down_proj,up_proj"
modules_to_save="embed_tokens,lm_head"
lora_dropout=0.05
pretrained_model=dnahlm-merge-hf
chinese_tokenizer_path=dnahlm-merge-hf
dataset_dir=sft_data
per_device_train_batch_size=8
per_device_eval_batch_size=8
gradient_accumulation_steps=8
output_dir=dnahlm-llama7b-sft
#peft_model=peft_model/dir
validation_file=val_data.json
deepspeed_config_file=ds_zero2_no_offload.json
torchrun --nnodes 1 --nproc_per_node 6 run_clm_sft_with_peft.py \
--deepspeed ${deepspeed_config_file} \
--model_name_or_path ${pretrained_model} \
--tokenizer_name_or_path ${chinese_tokenizer_path} \
--dataset_dir ${dataset_dir} \
--validation_split_percentage 0.001 \
--per_device_train_batch_size ${per_device_train_batch_size} \
--per_device_eval_batch_size ${per_device_eval_batch_size} \
--do_train \
--do_eval \
--seed $RANDOM \
--fp16 \
--num_train_epochs 3 \
--lr_scheduler_type cosine \
--learning_rate ${lr} \
--warmup_ratio 0.03 \
--weight_decay 0 \
--logging_strategy steps \
--logging_steps 10 \
--save_strategy steps \
--save_total_limit 3 \
--evaluation_strategy steps \
--eval_steps 100 \
--save_steps 200 \
--gradient_accumulation_steps ${gradient_accumulation_steps} \
--preprocessing_num_workers 4 \
--max_seq_length 256 \
--output_dir ${output_dir} \
--overwrite_output_dir \
--ddp_timeout 30000 \
--logging_first_step True \
--lora_rank ${lora_rank} \
--lora_alpha ${lora_alpha} \
--trainable ${lora_trainable} \
--modules_to_save ${modules_to_save} \
--lora_dropout ${lora_dropout} \
--torch_dtype float16 \
--validation_file ${validation_file} \
--gradient_checkpointing \
--ddp_find_unused_parameters False
04-gene-sft/1-finetue-intro.ipynb
+433
View File
@@ -0,0 +1,433 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "32216f81-0979-4afd-8c8c-16729cd0dab6",
"metadata": {},
"source": [
"# 4.1 模型微调VS指令微调"
]
},
{
"cell_type": "markdown",
"id": "7cd7f9b2-c0a3-48c2-848e-a1e9c7188f03",
"metadata": {},
"source": [
"## 一个典型的知乎问题\n",
"\n",
"### **问题**\n",
"\n",
"用LLM实现文本二分类，微调base模型还是微调chat模型比较好？[问题](https://www.zhihu.com/question/632473480/answer/38930949853)\n",
"\n",
"我想用开源LLM（例如chatglm，baichuan）实现文本二分类（比如正负情感分类），有一组训练数据可以用于微调模型，提升分类性能，这时候应该选择base模型还是chat模型？\n",
"\n",
"\n",
"### **回答**\n",
"1 如果是使用2分类的header，base模型好一些。\n",
"\n",
"也就是使用如下类似的的设置。\n",
"\n",
"model = AutoModelForSequenceClassification.from_pretrained(\n",
"\"yuanzhoulvpi/gpt2_chinese\", num_labels=2\n",
")\n",
"\n",
"对应的训练数据一般是这样的：\n",
"\n",
"| seq | label |\n",
"|------------------------------|-------|\n",
"| 他家的奶茶超级好喝。。。 | 1 |\n",
"| 他家的奶茶超级难喝。。。 | 0 |\n",
"\n",
"\n",
"2 如果是把分类问题，改成指令微调的模式，就是像\n",
"\n",
"```\n",
"{\n",
"\n",
"\"instruction\": \"你现在在做一项情感分类的任务，如果是积极情感，则回答积极。消极情感则回答消极。\"\n",
"\"input\"：他家的奶茶超级好喝。。。\n",
"\"output\"：“积极”\n",
"\n",
"}\n",
"```\n",
"\n",
"然后进行指令微调，lora/peft调整部分参数就行，一般是chat模型比较好。\n",
"\n",
"\n",
"\n",
"这种二分类问题，用llm就是大材小用了，一般就是选个小的的模型，用AutoModelForSequenceClassification效果最好，如果追求SOTA，有些研究表明搞成指令微调模式效果可能更好。"
]
},
{
"cell_type": "markdown",
"id": "2cfcc1e9-ddda-4a1c-871b-0508fd421ed5",
"metadata": {},
"source": [
"## 大模型微调（Fine-tuning）和指令微调（Instruction Tuning）\n",
"\n",
"普通的大模型微调（Fine-tuning）和指令微调（Instruction Tuning）是两种不同的训练方法，它们适用于不同的应用场景，并且在实现细节上也有所区别。\n",
"\n",
"\n",
"#### 1. **定义**\n",
"\n",
"普通微调是指在一个预训练好的大模型基础上，针对特定任务添加一个或多个新层（通常称为头部或 header），然后使用特定任务的数据集对整个模型（包括新添加的层）进行再训练的过程。对于分类任务，常见的做法是在 GPT-2 的顶部添加一个分类头。\n",
"\n",
"#### 2. **具体步骤**\n",
"\n",
"- **添加分类头**：为 GPT-2 添加一个分类头，该头通常包含线性层（全连接层）以及可能的激活函数和归一化层。\n",
" \n",
"- **准备数据**：准备好用于微调的任务特定数据集，如文本分类、情感分析等。\n",
" \n",
"- **微调过程**：\n",
" - 使用任务特定的数据集对整个模型（包括预训练权重和新添加的分类头）进行再训练。\n",
" - 通常会调整学习率、批量大小等超参数以优化性能。\n",
" - 可能只对新添加的层进行训练，或者对整个模型进行微调（取决于资源和需求）。\n",
"\n",
"#### 3. **适用场景**\n",
"\n",
"- **任务明确**：当有清晰的任务目标时，例如文本分类、命名实体识别等。\n",
"- **标签数据可用**：拥有足够的标注数据来进行监督学习。\n",
"\n",
"#### 4. **优点**\n",
"\n",
"- **针对性强**：能够有效地提升模型在特定任务上的表现。\n",
"- **资源利用效率高**：相比于从头开始训练，微调需要的计算资源和时间较少。\n",
"\n",
"#### 5. **缺点**\n",
"\n",
"- **泛化能力有限**：微调后的模型可能在未见过的任务或领域中表现不佳。\n",
"\n",
"### 指令微调（Instruction Tuning）\n",
"\n",
"#### 1. **定义**\n",
"\n",
"指令微调是一种更为通用的微调方法，它旨在让模型理解和遵循自然语言指令，而不是直接针对某个特定任务进行优化。这种方法通过提供一系列指令-输出对来训练模型，使其学会根据指令生成适当的响应。\n",
"\n",
"#### 2. **具体步骤**\n",
"\n",
"- **构造指令数据集**：创建一个包含各种指令及其预期输出的数据集。这些指令可以覆盖多种任务类型，如问答、翻译、摘要生成等。\n",
" \n",
"- **微调过程**：\n",
" - 使用指令数据集对模型进行训练，使模型能够理解并执行不同类型的指令。\n",
" - 强调模型对自然语言指令的理解和执行，而非特定于某一任务的优化。\n",
"\n",
"#### 3. **适用场景**\n",
"\n",
"- **多任务适应**：当希望模型能够在多种不同类型的任务中表现出色时。\n",
"- **少样本学习**：在仅有少量示例的情况下，仍然可以让模型快速适应新任务。\n",
"\n",
"#### 4. **优点**\n",
"\n",
"- **灵活性高**：模型可以在没有额外训练的情况下处理新的任务。\n",
"- **跨领域泛化能力强**：更有可能在未曾见过的任务或领域中保持良好的性能。\n",
"\n",
"#### 5. **缺点**\n",
"\n",
"- **复杂度增加**：指令微调通常涉及更多的训练数据和更复杂的训练过程。\n",
"- **评估难度较大**：由于任务的多样性，评估模型性能变得更加困难。\n",
"\n",
"\n",
"### 小结\n",
"\n",
"普通微调侧重于提高模型在特定任务上的性能，而指令微调则更加注重模型对自然语言指令的理解和执行能力。选择哪种方法取决于你的具体需求和应用场景。如果你有一个明确的任务并且有大量的标注数据，那么普通微调可能是更好的选择；如果你希望模型具有更高的灵活性和跨任务适应能力，则可以考虑指令微调。"
]
},
{
"cell_type": "markdown",
"id": "6203be53-18a5-447d-9071-32e031934b9c",
"metadata": {},
"source": [
"## 从GPT到chatGPT\n",
"\n",
"关键点在于指令微调（Instruction Tuning）\n",
"* 将所有任务统一为指令形式\n",
"* 多任务精调\n",
"* 与人类对齐（多样性）\n",
"* 进一步分为有监督指令微调和带有人类反馈的强化学习(RLHF)\n",
"\n",
"告别微调\n",
"\n",
"因为GPT-3使用了天量级的数据来进行预训练，所以学到的知识也更多更通用，以致于GPT-3打出的口号就是“告别微调的GPT-3”。\n",
"\n",
"相比于BERT这种预训练+微调的两阶段模型，GPT-3的目标是模型更加通用，从而解决BERT这种下游任务微调需要依赖领域标注数据的情况。\n",
"\n",
"拿我们实际业务举例，我主要做分本分类任务。对于使用BERT来完成文本分类任务来说，首先我需要使用海量的无标注文本数据进行预训练学习语言学知识。\n",
"\n",
"幸运的是这种预训练过程一般是一次性的，训练完成后可以把模型保存下来继续使用。很多大厂比如谷歌、Facebook等把得到的预训练模型开源了出来，所以咱们只需要导入预训练好的模型权重就可以直接使用了，相当于完成了模型的预训练过程；第二阶段就是微调了，对于文本分类等下游任务来说， 我们需要一批带标签的训练语料来微调模型。不同的下游任务会需要特定的训练语料。这时候面临的一个最大的问题是训练语料是需要人工标注的，而标注的成本是非常高的。除此之外不同的标注人员因为经验阅历等不同导致对同一条文本的理解也不同，所以容易出现标注不一致的问题。当标注数据量较少时还容易出现模型过拟合。归根结底就是微调是需要标注数据的，而获取标注数据的成本是很高的。\n",
"\n",
"为了解决这个问题，GPT-3可以让NLPer不用标注训练语料就能很好的完成下游任务，让GPT-3更通用更便利。GPT-3不需要进行微调的结构图如下所示：\n",
"\n",
"<img src='img/sft.png' width='600px' />"
]
},
{
"cell_type": "markdown",
"id": "28e037df-734b-4fe7-ac07-311f1b3a7d7b",
"metadata": {},
"source": [
"## 指令微调数据构建\n",
"\n",
"<img src='img/sft2.png' width='800px' />\n",
"\n",
"\n",
"\n",
"根据典型的分类语料数据，构建指令微调数据\n",
"\n",
"目前如llama等都使用Alpaca格式\n",
"\n",
"指令数据当做一般的文本，进行无监督的训练，和预训练流程一致"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "64312191-423f-4a18-aa0c-036374e93fb2",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"import os\n",
"# 设置环境变量, autodl一般区域\n",
"result = subprocess.run('bash -c \"source /etc/network_turbo && env | grep proxy\"', shell=True, capture_output=True, text=True)\n",
"output = result.stdout\n",
"for line in output.splitlines():\n",
" if '=' in line:\n",
" var, value = line.split('=', 1)\n",
" os.environ[var] = value"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "32c16282-f9f1-4545-b522-daf2b39b4ead",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
")"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#原始模型\n",
"from transformers import AutoModel\n",
"model = AutoModel.from_pretrained(\"gpt2\")\n",
"model"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "1149163f-4d89-472e-8d45-ebcbb5f9575e",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of GPT2ForSequenceClassification were not initialized from the model checkpoint at gpt2 and are newly initialized: ['score.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
]
},
{
"data": {
"text/plain": [
"GPT2ForSequenceClassification(\n",
" (transformer): GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" )\n",
" (score): Linear(in_features=768, out_features=2, bias=False)\n",
")"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#分类微调模型\n",
"from transformers import AutoModelForSequenceClassification\n",
"ft_model = AutoModelForSequenceClassification.from_pretrained(\"gpt2\", num_labels=2)\n",
"ft_model"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "09735059-507c-48c4-893f-ca0da21ce5e8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GPT2LMHeadModel(\n",
" (transformer): GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" )\n",
" (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
")"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#指令微调模型\n",
"from transformers import AutoModelForCausalLM\n",
"sft_model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n",
"sft_model"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d1407cbe-4996-4898-a135-e26d28da2a2a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"GPT2LMHeadModel(\n",
" (transformer): GPT2Model(\n",
" (wte): Embedding(50257, 768)\n",
" (wpe): Embedding(1024, 768)\n",
" (drop): Dropout(p=0.1, inplace=False)\n",
" (h): ModuleList(\n",
" (0-11): 12 x GPT2Block(\n",
" (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (attn): GPT2SdpaAttention(\n",
" (c_attn): Conv1D(nf=2304, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=768)\n",
" (attn_dropout): Dropout(p=0.1, inplace=False)\n",
" (resid_dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" (mlp): GPT2MLP(\n",
" (c_fc): Conv1D(nf=3072, nx=768)\n",
" (c_proj): Conv1D(nf=768, nx=3072)\n",
" (act): NewGELUActivation()\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
" )\n",
" (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
")"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import GPT2LMHeadModel\n",
"gpt2_model = GPT2LMHeadModel.from_pretrained(\"gpt2\")\n",
"gpt2_model"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "92fc8e55-2d90-4694-b8df-90885d08d51a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
04-gene-sft/2-gpt2-instruction-ft.ipynb
+1688
View File
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More