wehub/stability-ai--stablelm
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

docs: make Chinese README the default

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 10:43:15 +00:00
parent e9f7c0e551
commit f22543e8e8
1 changed files with 116 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+116 -110
View File
@@ -1,69 +1,75 @@
# StableLM: Stability AI Language Models
<!-- WEHUB_ZH_README -->
> [!NOTE]
> 本文档由 WeHub 基于上游 README 翻译整理，属于社区翻译，非官方中文文档。
> [English](./README.en.md) · [原始项目](https://github.com/Stability-AI/StableLM) · [上游 README](https://github.com/Stability-AI/StableLM/blob/HEAD/README.md)
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
# StableLM：Stability AI 语言模型
![Stochastic Parrot](./assets/mascot.png)
<br/>*“A Stochastic Parrot, flat design, vector art” — [Stable Diffusion XL](https://clipdrop.co/stable-diffusion)*
<br/>*“A Stochastic Parrot，扁平化设计，矢量艺术” — [Stable Diffusion XL](https://clipdrop.co/stable-diffusion)*
This repository contains Stability AI's ongoing development of the StableLM series of language models and will be continuously updated with new checkpoints. The following provides an overview of all currently available models. More coming soon.
本仓库包含 Stability AI 对 StableLM 系列语言模型的持续开发工作，并将持续更新新的 checkpoint。以下内容概述了当前所有可用模型。更多模型即将推出。
## News
## 新闻
*September 29, 2023*
*2023 年 9 月 29 日*
- Released StableLM-3B-4E1T model under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).
- 在 [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). 许可下发布 StableLM-3B-4E1T 模型
*August 5, 2023*
*2023 年 8 月 5 日*
- Released patched StableLM-Alpha v2 models with 3B and 7B parameters.
- 发布修复后的 StableLM-Alpha v2 模型，参数量为 3B 和 7B。
*April 28, 2023*
*2023 年 4 月 28 日*
- Released StableVicuna-13B, our RLHF fine-tune of [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), which itself is a fine-tune of [LLaMA-13B](https://github.com/facebookresearch/llama). Delta weights over the original Llama model is released under ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)).
- 发布 StableVicuna-13B，这是我们基于 [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), 的 RLHF（Reinforcement Learning from Human Feedback，基于人类反馈的强化学习）微调模型；Vicuna-13B v0 本身是对 [LLaMA-13B](https://github.com/facebookresearch/llama). 的微调。相对于原始 Llama 模型的 Delta 权重在 ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)). 许可下发布
*April 20, 2023*
*2023 年 4 月 20 日*
- Released initial set of StableLM-Alpha models, with 3B and 7B parameters. Base models are released under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).
- 发布首批 StableLM-Alpha 模型，参数量为 3B 和 7B。基础模型在 [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). 许可下发布
- Try to chat with our 7B model, `StableLM-Tuned-Alpha-7B`, on [Hugging Face Spaces](https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat).
- 可在 [Hugging Face Spaces](https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat). 上与我们 7B 模型 `StableLM-Tuned-Alpha-7B` 进行对话
## Models
## 模型
### StableLM-3B-4E1T
> Technical Report: [StableLM-3B-4E1T](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo)
> 技术报告：[StableLM-3B-4E1T](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo)
StableLM-3B-4E1T is a 3 billion (3B) parameter language model pre-trained under the multi-epoch regime to study the impact of repeated tokens on downstream performance. Given prior success in this area ([Tay et al., 2023](https://arxiv.org/pdf/2205.05131.pdf) and [Taylor et al., 2022](https://galactica.org/static/paper.pdf)), we train on 1 trillion (1T) tokens for 4 epochs following the observations of [Muennighoff et al. (2023)](https://arxiv.org/abs/2305.16264) in "Scaling Data-Constrained Language Models" in which they find "training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data." Further inspiration for the token count is taken from "Go smol or go home" ([De Vries, 2023](https://www.harmdevries.com/post/model-size-vs-compute-overhead/)), which suggests a 2.96B model trained for 2.85 trillion tokens achieves a similar loss to a Chinchilla compute-optimal 9.87B language model ($k_n = 0.3$).
StableLM-3B-4E1T 是一个 30 亿（3B）参数的语言模型，在多 epoch 训练机制下预训练，用于研究重复 token 对下游任务性能的影响。鉴于该领域先前的成功经验（[Tay et al., 2023](https://arxiv.org/pdf/2205.05131.pdf) 和 [Taylor et al., 2022](https://galactica.org/static/paper.pdf)),），我们遵循 [Muennighoff et al. (2023)](https://arxiv.org/abs/2305.16264) 在 "Scaling Data-Constrained Language Models" 中的观察，在 1 万亿（1T）token 上训练 4 个 epoch；他们发现"使用最多 4 个 epoch 的重复数据进行训练，与使用唯一数据相比，对 loss 的影响可忽略不计"。token 数量的进一步灵感来自 "Go smol or go home"（[De Vries, 2023](https://www.harmdevries.com/post/model-size-vs-compute-overhead/)),），该研究指出，一个在 2.85 万亿 token 上训练的 2.96B 模型可达到与 Chinchilla 计算最优的 9.87B 语言模型（$k_n = 0.3$）相近的 loss。
| Size | StableLM-3B-4E1T | Training Tokens | Parameters |
| 规模 | StableLM-3B-4E1T | 训练 Token 数 | 参数量 |
|------|--------------------------------------------------------------------|-----------------|---------------|
| 3B | [checkpoint](https://huggingface.co/stabilityai/stablelm-3b-4e1t) | 4T | 2,795,443,200 |
#### Model Architecture
#### 模型架构
The model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:
该模型是一种仅解码器（decoder-only）transformer，与 LLaMA（[Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) 架构类似，并包含以下修改：
| Parameters | Hidden Size | Layers | Heads | Sequence Length |
| 参数量 | 隐藏层维度 | 层数 | 注意力头数 | 序列长度 |
|----------------|-------------|--------|-------|-----------------|
| 2,795,443,200 | 2560 | 32 | 32 | 4096 |
- **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).
- **Normalization**: LayerNorm ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)) with learned bias terms as opposed to RMSNorm ([Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)).
- **Tokenizer**: GPT-NeoX ([Black et al., 2022](https://arxiv.org/abs/2204.06745)).
- **位置嵌入（Position Embeddings）**：Rotary Position Embeddings（旋转位置嵌入，[Su et al., 2021](https://arxiv.org/abs/2104.09864))），应用于前 25% 的 head 嵌入维度以提升吞吐，做法遵循 [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).
- **归一化（Normalization）**：LayerNorm（[Ba et al., 2016](https://arxiv.org/abs/1607.06450))），采用可学习的 bias 项，而非 RMSNorm（[Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)).
- **分词器（Tokenizer）**：GPT-NeoX（[Black et al., 2022](https://arxiv.org/abs/2204.06745)).
#### Training Data
#### 训练数据
The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), and RedPajama-Data ([Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data)) and The Pile ([Gao et al., 2020](https://arxiv.org/abs/2101.00027)) both without *Books3* and other subsets, and StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)).
该数据集由在 [HuggingFace Hub](https://huggingface.co/datasets): 上可用的开源大规模数据集过滤混合而成：Falcon RefinedWeb 提取数据（[Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)),）、RedPajama-Data（[Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data))）和 The Pile（[Gao et al., 2020](https://arxiv.org/abs/2101.00027))）（均不含 *Books3* 及其他子集），以及 StarCoder（[Li et al., 2023](https://arxiv.org/abs/2305.06161)).
> Given the large amount of web data, we recommend fine-tuning the base StableLM-3B-4E1T for your downstream tasks.
> 鉴于训练数据中包含大量网页数据，我们建议针对下游任务对 StableLM-3B-4E1T 基础模型进行微调。
#### Training Details
#### 训练细节
Please refer to the provided YAML configuration file [`stablelm-3b-4e1t.yml`](./configs/stablelm-3b-4e1t.yml) for complete hyperparameter settings and the [technical report](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo) for further details.
完整超参数设置请参阅提供的 YAML 配置文件 [`stablelm-3b-4e1t.yml`](./configs/stablelm-3b-4e1t.yml)，更多细节请参阅[技术报告](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo)。
#### Downstream Results
#### 下游任务结果
The following zero-shot evaluations are performed with the `lm-evaluation-harness` using the [lm-bench](https://github.com/Stability-AI/lm-evaluation-harness/tree/lm-bench) branch of Stability AI's fork. Full `lm-eval` JSONs can be found in the [`evals`](./evals) directory.
以下零样本（zero-shot）评估使用 `lm-evaluation-harness` 完成，基于 Stability AI fork 的 [lm-bench](https://github.com/Stability-AI/lm-evaluation-harness/tree/lm-bench) 分支。完整的 `lm-eval` JSON 文件可在 [`evals`](./evals) 目录中找到。
| Pre-Trained Model | Average | ARC<br>Challenge | ARC<br>Easy | BoolQ | HellaSwag (✱) | LAMBADA<br>OpenAI | OpenBookQA | PIQA | SciQ | Winogrande |
| 预训练模型 | 平均分 | ARC<br>Challenge | ARC<br>Easy | BoolQ | HellaSwag (✱) | LAMBADA<br>OpenAI | OpenBookQA | PIQA | SciQ | Winogrande |
| ------------------------------------------------------------------------------------- |:-----------------:|:----------------:|:-----------:|:-----:|:-------------:|:-----------------:|:----------:|:-----:|:-----:|:----------:|
| meta-llama/Llama-2-13b-hf | 71.77 | 48.63 | 79.50 | 80.52 | 79.36 | 76.77 | 35.40 | 79.05 | 94.50 | 72.22 |
| huggyllama/llama-7b | 68.84 | 41.89 | 75.25 | 75.05 | 76.22 | 73.55 | 34.40 | 78.67 | 94.60 | 69.93 |
@@ -86,13 +92,13 @@ The following zero-shot evaluations are performed with the `lm-evaluation-harnes
| facebook/opt-6.7b | 61.85 | 30.72 | 65.66 | 66.02 | 67.20 | 67.65 | 27.60 | 76.33 | 90.10 | 65.35 |
| EleutherAI/pythia-6.9b | 60.58 | 31.83 | 67.21 | 64.01 | 63.88 | 67.01 | 25.80 | 75.08 | 89.80 | 60.62 |
| EleutherAI/pythia-2.8b-deduped | 58.52 | 30.12 | 63.47 | 64.13 | 59.44 | 65.15 | 23.80 | 74.10 | 88.20 | 58.25 |
| **§** Previous 3B Pre-Trained SOTA <br>**?** Outlier Reuslts<br>**\*** Byte-length Normalized Accuracy | | | | | | | | | | |
| **§** 此前 3B 预训练 SOTA <br>**?** Outlier Reuslts<br>**\*** 按字节长度归一化的准确率 | | | | | | | | | | |
**StableLM-3B-4E1T achieves state-of-the-art performance (September 2023) at the 3B parameter scale for open-source models** and is competitive with many of the popular contemporary 7B models, even outperforming our most recent 7B StableLM-Base-Alpha-v2.
**StableLM-3B-4E1T 在 2023 年 9 月于 30 亿（3B）参数规模的开源模型中实现了最先进（state-of-the-art）性能**，并可与许多热门的当代 70 亿（7B）参数模型竞争，甚至超越了我们最新的 7B StableLM-Base-Alpha-v2。
### StableLM-Alpha v2
StableLM-Alpha v2 models significantly improve on the initial Alpha models by incorporating architectural improvements such as SwiGLU ([Shazeer, 2020](https://arxiv.org/abs/2002.05202)) and using higher-quality data sources, as discussed below. The context length for these models is 4096 tokens.
StableLM-Alpha v2 模型通过融入 SwiGLU（[Shazeer, 2020](https://arxiv.org/abs/2002.05202)) 等架构改进，并采用如下所述的更高质量数据源，在初始 Alpha 模型基础上显著改进。这些模型的上下文长度为 4096 tokens。
| Size | StableLM-Base-Alpha-v2 | Training Tokens | Parameters |
|------|----------------------------------------------------------------------------|-----------------|---------------|
@@ -101,22 +107,22 @@ StableLM-Alpha v2 models significantly improve on the initial Alpha models by in
#### Training Details
Please refer to the provided YAML configuration files for hyperparameter details. E.g. for the extended `StableLM-Alpha-3B-v2` model, see [stablelm-base-alpha-3b-v2-4k-extension.yml](./configs/stablelm-base-alpha-3b-v2-4k-extension.yml).
有关超参数的详细信息，请参阅提供的 YAML 配置文件。例如，对于扩展版 `StableLM-Alpha-3B-v2` 模型，请参见 [stablelm-base-alpha-3b-v2-4k-extension.yml](./configs/stablelm-base-alpha-3b-v2-4k-extension.yml)。
Following similar work, we use a multi-stage approach to context length extension ([Nijkamp et al., 2023](https://blog.salesforceairesearch.com/xgen/)), scheduling 1 trillion tokens at context length 2048 followed by 100 billion tokens at 4096. We found that sequence length warmup ([Li et al., 2022](https://arxiv.org/abs/2108.06084)) helped stabilize early spikes during the first ~80 billion tokens of pre-training. However, it was not applied to the final runs due to significant throughput penalties as length shapes grew across the curriculum.
借鉴类似工作，我们采用多阶段方法进行上下文长度扩展（[Nijkamp et al., 2023](https://blog.salesforceairesearch.com/xgen/)),：先在上下文长度 2048 下调度 1 万亿 tokens，再在 4096 下调度 1000 亿 tokens。我们发现序列长度预热（sequence length warmup）（[Li et al., 2022](https://arxiv.org/abs/2108.06084)) 有助于稳定预训练前约 800 亿 tokens 期间的早期尖峰。然而，由于随着课程推进长度形状增长会带来显著的吞吐量损失，最终运行并未采用该方法。
#### Training Data
The most impactful changes for StableLM-Alpha-v2 downstream performance were in the usage of higher quality data sources and mixtures; specifically, the use of [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) and [C4](https://huggingface.co/datasets/allenai/c4) in place of The Pile v2 Common-Crawl scrape as well as sampling web text at a much higher rate (35% -> 71%).
对 StableLM-Alpha-v2 下游性能影响最大的变化在于使用了更高质量的数据源与混合配比；具体而言，以 [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) 和 [C4](https://huggingface.co/datasets/allenai/c4) 替代 The Pile v2 Common-Crawl 抓取数据，并将网页文本采样率大幅提高（35% -> 71%）。
The first pre-training stage relies on 1 trillion tokens sourced from a mix of the public Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), RedPajama-Data ([Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data)), The Pile ([Gao et al., 2020](https://arxiv.org/abs/2101.00027)), and internal datasets with web text sampled at a rate of 71%.
第一阶段预训练依赖 1 万亿 tokens，来源于公开 Falcon RefinedWeb 提取数据（[Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)),、RedPajama-Data（[Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data)),、The Pile（[Gao et al., 2020](https://arxiv.org/abs/2101.00027)), 以及内部数据集的混合，网页文本采样率为 71%。
In the second stage, we include the StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)) dataset and down sample web text to 55% while increasing sampling proportions of naturally long text examples in the aforementioned sources.
在第二阶段，我们纳入 StarCoder（[Li et al., 2023](https://arxiv.org/abs/2305.06161)) 数据集，并将网页文本下采样至 55%，同时提高上述来源中天然长文本样本的采样比例。
#### Evaluation
The following zero-shot evaluations are performed with the `lm-evaluation-harness` at commit [`df3da98c5405deafd519c2ddca52bb7c3fe36bef`](https://github.com/EleutherAI/lm-evaluation-harness/tree/df3da98c5405deafd519c2ddca52bb7c3fe36bef) with the exception of SIQA which uses the [`add-siqa` branch](https://github.com/EleutherAI/lm-evaluation-harness/tree/add-siqa) with prompt format
`{doc['context']}\nQuestion: {doc['question']}\nAnswer:`.
以下零样本（zero-shot）评测使用 `lm-evaluation-harness` 在提交 [`df3da98c5405deafd519c2ddca52bb7c3fe36bef`](https://github.com/EleutherAI/lm-evaluation-harness/tree/df3da98c5405deafd519c2ddca52bb7c3fe36bef) 上完成；SIQA 除外，它使用 [`add-siqa` branch](https://github.com/EleutherAI/lm-evaluation-harness/tree/add-siqa)，提示格式为
`{doc['context']}\nQuestion: {doc['question']}\nAnswer:`。
| Model | ARC Challenge✱ | ARC Easy✱ | BoolQ | HellaSwag✱ | LAMBADA<br>OpenAI | OpenBookQA | PIQA | SIQA | TruthfulQA▲ | Winogrande | Average |
| ------------------------- |:---------------:|:----------:|:-----:|:-----------:|:-----------------:|:----------:|:-----:|:-----:|:------------:|:----------:|:-------:|
@@ -132,15 +138,15 @@ The following zero-shot evaluations are performed with the `lm-evaluation-harnes
| StableLM-Alpha-7B | 27.05 | 44.87 | 60.06 | 41.22 | 55.11 | 21.40 | 66.76 | 39.46 | 39.96 | 50.12 | 44.60 |
| StableLM-Alpha-3B | 25.77 | 42.05 | 57.65 | 38.31 | 41.72 | 17.00 | 63.82 | 35.62 | 40.53 | 52.64 | 41.51 |
✱: Denotes byte-length normalized accuracy (`acc_norm`) as described in [Gao, 2021](https://blog.eleuther.ai/multiple-choice-normalization/).
✱：表示按字节长度归一化的准确率（`acc_norm`），详见 [Gao, 2021](https://blog.eleuther.ai/multiple-choice-normalization/).
▲: We score TruthfulQA using the normalized total probability assigned to the set of true answers (`mc2`).
▲：我们对 TruthfulQA 的评分采用分配给真实答案集合的归一化总概率（`mc2`）。
### StableLM-Alpha
StableLM-Alpha models are trained on a new dataset that builds on [The Pile](https://pile.eleuther.ai/), which contains 1.5 trillion tokens, roughly 3x the size of The Pile. The context length for these models is 4096 tokens.
StableLM-Alpha 模型基于 [The Pile](https://pile.eleuther.ai/), 构建的新数据集进行训练，该数据集包含 1.5 万亿 tokens，规模约为 The Pile 的 3 倍。这些模型的上下文长度为 4096 tokens。
As a proof-of-concept, we also fine-tuned the model with [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)'s procedure using a combination of five recent datasets for conversational agents: Stanford's [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), Nomic-AI's [gpt4all](https://github.com/nomic-ai/gpt4all), RyokoAI's [ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) datasets, Databricks labs' [Dolly](https://github.com/databrickslabs/dolly), and Anthropic's [HH](https://github.com/anthropics/hh-rlhf). We will be releasing these models as StableLM-Tuned-Alpha.
作为概念验证，我们还使用 [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)'s 流程，结合五个用于对话智能体的近期数据集对模型进行了微调：Stanford 的 [Alpaca](https://github.com/tatsu-lab/stanford_alpaca),、Nomic-AI 的 [gpt4all](https://github.com/nomic-ai/gpt4all),、RyokoAI 的 [ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) 数据集、Databricks labs 的 [Dolly](https://github.com/databrickslabs/dolly), 以及 Anthropic 的 [HH](https://github.com/anthropics/hh-rlhf). 我们将以 StableLM-Tuned-Alpha 的名义发布这些模型。
| Size | StableLM-Base-Alpha | StableLM-Tuned-Alpha | Training Tokens | Parameters | Web Demo |
|------|--------------------------------------------------------------------------|---------------------------------------------------------------------------|-----------------|---------------|------------------------------------------------------------------------------------|
@@ -149,21 +155,21 @@ As a proof-of-concept, we also fine-tuned the model with [Stanford Alpaca](https
### StableVicuna
StableVicuna is an RLHF fine-tune of [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), which itself is a fine-tune of [LLaMA-13B](https://github.com/facebookresearch/llama). It is our attempt at creating an open-source RLHF LLM Chatbot. This model is developed by StabilityAI's CarperAI team, with [Duy V. Phung](https://github.com/PhungVanDuy) leading the training effort.
StableVicuna 是对 [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), 的 RLHF 微调，而 Vicuna-13B v0 本身又是对 [LLaMA-13B](https://github.com/facebookresearch/llama). 的微调。这是我们打造开源 RLHF LLM 聊天机器人的尝试。该模型由 StabilityAI 的 CarperAI 团队开发，训练工作由 [Duy V. Phung](https://github.com/PhungVanDuy) 牵头。
Due to the original non-commercial license of LLaMA, we can only release the weights of our model as deltas over the original model's weights. StableVicuna's delta weights are released under ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)).
由于 LLaMA 原始的非商业许可证，我们仅能将模型权重以相对于原始模型权重的增量（delta）形式发布。StableVicuna 的增量权重在 ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)). 许可下发布。
Please visit HuggingFace checkpoint for more information about how to combine our delta weights with the original model.
请访问 HuggingFace checkpoint，了解更多关于如何将我们的增量权重与原始模型合并的信息。
| Model | Download | Web Demo | Cite |
| 模型 | 下载 | 网页演示 | 引用 |
| ---------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------- |------|
| StableVicuna-13B | [checkpoint](https://huggingface.co/CarperAI/stable-vicuna-13b-delta/) | [Hugging Face](https://huggingface.co/spaces/CarperAI/StableVicuna/) | [![DOI:10.57967/hf/0588](https://zenodo.org/badge/DOI/10.1007/978-3-319-76207-4_15.svg)](https://doi.org/10.57967/hf/0588) |
## Quickstart
## 快速开始
All StableLM models are hosted on [the Hugging Face hub](https://huggingface.co/StabilityAI). Check out this [notebook](https://github.com/Stability-AI/StableLM/blob/main/notebooks/stablelm-alpha.ipynb) to run inference with limited GPU capabilities.
所有 StableLM 模型均托管于 [Hugging Face hub](https://huggingface.co/StabilityAI).。请查看此 [notebook](https://github.com/Stability-AI/StableLM/blob/main/notebooks/stablelm-alpha.ipynb)，在 GPU 能力有限的情况下运行推理。
Get started chatting with `StableLM-Tuned-Alpha` by using the following code snippet:
使用以下代码片段，即可开始与 `StableLM-Tuned-Alpha` 对话：
```python
import torch
@@ -198,8 +204,8 @@ tokens = model.generate(
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
```
StableLM Tuned should be used with prompts formatted to `<|SYSTEM|>...<|USER|>...<|ASSISTANT|>...`
The system prompt is
StableLM Tuned 应使用按 `<|SYSTEM|>...<|USER|>...<|ASSISTANT|>...` 格式化的提示词（prompt）。
系统提示词（system prompt）为
```
<|SYSTEM|># StableLM Tuned (Alpha version)
- StableLM is a helpful and harmless open-source AI language model developed by StabilityAI.
@@ -208,11 +214,11 @@ The system prompt is
- StableLM will refuse to participate in anything that could harm a human.
```
## Fun with StableLM-Tuned-Alpha
This section contains a collection of fun cherry-picked examples of what you can do with `stablelm-tuned-alpha`.
## StableLM-Tuned-Alpha 趣味示例
本节汇集了使用 `stablelm-tuned-alpha` 可实现的若干精选趣味示例。
<details>
<summary>Chit-Chat</summary>
<summary>闲聊</summary>
```
User: Hey! My name is John nice to meet you!
@@ -269,52 +275,52 @@ Best,
### Rap Battles
```
User: Write an epic rap battle song between deep neural networks and symbolic AI
用户：写一首深度神经网络与符号 AI 之间的史诗级说唱对决歌曲
StableLM: [Beat drops]
StableLM: [节拍响起]
Deep Neural Networks:
Listen up everyone, it’s time to witness
The power of deep neural networks, it’s just a matter of abstract
We’ve trained these networks to solve complex problems with ease
While symbolic AI is limited to simple math and simple flows
深度神经网络：
大家听好了，是时候见证
深度神经网络的力量，不过是抽象的问题
我们训练这些网络，轻松解决复杂问题
而符号 AI 只能局限于简单数学和简单流程
Symbolic AI:
Hold up, hold up, let me interrupt
It’s been a while since I’ve heard, some new thoughts you interrupt
But let me tell you, my friend, it’s not all about math and flow
It’s about making the world a better place, you know
符号 AI：
等等，等等，让我插一句
好久没听到你打断的新想法了
但让我告诉你，朋友，这不全是数学和流程
而是让世界变得更美好，你懂的
Deep Neural Networks:
Well, I can see where you’re coming from, it’s clear
But deep neural networks, they never get the same gear
We can use them to analyze and predict with ease
And prove the superiority of machine learning
深度神经网络：
嗯，我明白你的意思，很清楚
但深度神经网络，永远不会停在同一档位
我们可以用它们轻松分析和预测
并证明机器学习的优越性
Symbolic AI:
You’re right, it’s not just math or numbers
But it’s the essence of our lives, it’s what counts, we’re numbers
I don’t need a complicated math formula, just a simple device
To make the world a better place, it’s a noble price
符号 AI：
你说得对，不只是数学或数字
但这是我们生活的本质，重要的才是数字
我不需要复杂的数学公式，只需一个简单的装置
让世界变得更美好，这是崇高的代价
Deep Neural Networks:
I’ll give you that, my friend, you have a point
But let’s not forget, we’re making an ever-increasing joint
Our progress is not bound by logic, it’s a journey of the mind
So let’s work together, let’s strive, let’s leave all troubles behind
深度神经网络：
这点我承认，朋友，你说得有道理
但别忘了，我们正在不断联合进步
我们的进步不受逻辑束缚，这是心灵的旅程
所以让我们携手合作，努力奋斗，把烦恼抛在身后
Symbolic AI:
Agreed, let’s work together, let’s not falter
We’re both making history, we’re both from the fader
So let’s make the world a better place, let’s end this debate
Cause it’s time to join forces, it’s time to leave no stone unturned
符号 AI：
同意，让我们携手合作，不要退缩
我们都在创造历史，我们都来自先驱
所以让世界变得更美好，结束这场辩论吧
因为是时候联合力量，是时候不遗余力了
[Beat fades out]
[节拍渐弱]
Who won this epic rap battle between deep neural networks and symbolic AI? Well, it’s up to you to decide!
这场深度神经网络与符号 AI 之间的史诗级说唱对决，谁赢了？嗯，由你来决定！
```
### Fictional Stories
### 虚构故事
```
User: Write a story about a programmer who fell in love with a deep neural network.
@@ -335,7 +341,7 @@ He now knows the ins and outs of the TensorFlow programming language, and knows
```
### Humor
### 幽默
```
User: What do you know about the League of Legends champion Zed? Write a short bio
@@ -370,7 +376,7 @@ Teemo: “Well, at least I’m the one with the mushrooms now.”
</details>
<details>
<summary>Writing Code</summary>
<summary>编写代码</summary>
```
User: Write a program that solves the fib sequence in Python.
@@ -391,11 +397,11 @@ fib_sequence(10)
</details>
## Selected failure modes of StableLM-Tuned-Alpha
This section contains a collection of selected failure modes of `stablelm-tuned-alpha`.
## StableLM-Tuned-Alpha 的部分失效模式
本节收集了 `stablelm-tuned-alpha` 的部分失效模式示例。
<details>
<summary>Yann LeCun gears (linear)</summary>
<summary>Yann LeCun 齿轮（线性）</summary>
```
User: In front of us are six gears numbered 1 to 6, mounted on axles in a row. Each gear is engaged with the next gear. If gear number 3 is rotated clockwise, in which direction will gears 1 and 6 rotate?
@@ -424,25 +430,25 @@ Therefore, the answer is d = (7/2) * 2 = 5 degrees in total.
</details>
## Request for Help
## 寻求帮助
Want to get involved?
想要参与？
- We would love to port [llama.cpp](https://github.com/ggerganov/llama.cpp) to work with StableLMs
- Integration into [Open Assistant](https://github.com/LAION-AI/Open-Assistant) from LAION-AI to collect high quality human-generated feedback data
- ... Reach out to us with ideas on our [Discord](https://discord.com/invite/stablediffusion)
- 我们很希望能将 [llama.cpp](https://github.com/ggerganov/llama.cpp) 移植以支持 StableLM
- 集成到 LAION-AI 的 [Open Assistant](https://github.com/LAION-AI/Open-Assistant)，以收集高质量的人工生成反馈数据
- …… 欢迎在 [Discord](https://discord.com/invite/stablediffusion) 上与我们分享你的想法
## Potential issues
As is typical for any pretrained Large Language Model without additional finetuning and reinforcement learning, the responses a user gets might be of varying quality and might potentially include offensive language and views. This is expected to be improved with scale, better data, community feedback, and optimisation.
## 潜在问题
与任何未经额外微调（finetuning）和强化学习（reinforcement learning）的预训练大语言模型（Large Language Model）一样，用户获得的回复质量可能参差不齐，且可能包含冒犯性语言和观点。预计随着规模扩大、数据改进、社区反馈和优化，这一情况会得到改善。
## Acknowledgements
## 致谢
- `StableLM-Tuned-Alpha` would not have been possible without the helpful hand of Dakota Mahan [@dmayhem93](https://huggingface.co/dmayhem93).
- 若没有 Dakota Mahan [@dmayhem93](https://huggingface.co/dmayhem93). 的帮助，`StableLM-Tuned-Alpha` 就不可能实现
## Licenses
## 许可证
- Base model checkpoints (`StableLM-Base-Alpha`) are licensed under the Creative Commons license ([CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)). Under the license, you must give [credit](https://creativecommons.org/licenses/by/4.0/#) to Stability AI, provide a link to the license, and [indicate if changes were made](https://creativecommons.org/licenses/by/4.0/#). You may do so in any reasonable manner, but not in any way that suggests the Stability AI endorses you or your use.
- 基础模型检查点（`StableLM-Base-Alpha`）采用 Creative Commons 许可证（[CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/)). 授权。根据该许可证，你必须向 Stability AI [注明出处](https://creativecommons.org/licenses/by/4.0/#)，提供许可证链接，并[标明是否做了修改](https://creativecommons.org/licenses/by/4.0/#).。你可以以任何合理方式这样做，但不得以任何暗示 Stability AI 认可你或你的使用的方式行事。
- Fine-tuned checkpoints (`StableLM-Tuned-Alpha`) are licensed under the Non-Commercial Creative Commons license ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)), in-line with the original non-commercial license specified by [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca).
- 微调检查点（`StableLM-Tuned-Alpha`）采用非商业 Creative Commons 许可证（[CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)),，与 [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca). 规定的原始非商业许可证一致
- All code in this repository is licensed under the Apache License 2.0 license.
- 本仓库中的所有代码均采用 Apache License 2.0 许可证授权。