mirrors/stability-ai--stablelm
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

release: add stablelm-3b-4e1t (#101)

Browse Source
This commit is contained in:
Jonathan Tow
2023-09-29 18:41:27 -04:00
committed by GitHub
parent 3404ef1257
commit 92f9c7a010
34 changed files with 1663 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+70 -3
View File
@@ -1,12 +1,16 @@
# StableLM: Stability AI Language Models
![Stochastic Parrot](/assets/mascot.png)
![Stochastic Parrot](./assets/mascot.png)
<br/>*“A Stochastic Parrot, flat design, vector art” — [Stable Diffusion XL](https://clipdrop.co/stable-diffusion)*
This repository contains Stability AI's ongoing development of the StableLM series of language models and will be continuously updated with new checkpoints. The following provides an overview of all currently available models. More coming soon.
## News
*September 29, 2023*
- Released StableLM-3B-4E1T model under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).
*August 5, 2023*
- Released patched StableLM-Alpha v2 models with 3B and 7B parameters.
@@ -23,9 +27,72 @@ This repository contains Stability AI's ongoing development of the StableLM seri
## Models
### StableLM-3B-4E1T
> Technical Report: [StableLM-3B-4E1T](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo)
StableLM-3B-4E1T is a 3 billion (3B) parameter language model pre-trained under the multi-epoch regime to study the impact of repeated tokens on downstream performance. Given prior success in this area ([Tay et al., 2023](https://arxiv.org/pdf/2205.05131.pdf) and [Taylor et al., 2022](https://galactica.org/static/paper.pdf)), we train on 1 trillion (1T) tokens for 4 epochs following the observations of [Muennighoff et al. (2023)](https://arxiv.org/abs/2305.16264) in "Scaling Data-Constrained Language Models" in which they find "training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data." Further inspiration for the token count is taken from "Go smol or go home" ([De Vries, 2023](https://www.harmdevries.com/post/model-size-vs-compute-overhead/)), which suggests a 2.96B model trained for 2.85 trillion tokens achieves a similar loss to a Chinchilla compute-optimal 9.87B language model ($k_n = 0.3$).
| Size | StableLM-3B-4E1T | Training Tokens | Parameters |
|------|--------------------------------------------------------------------|-----------------|---------------|
| 3B | [checkpoint](https://huggingface.co/stabilityai/stablelm-3b-4e1t) | 4T | 2,795,443,200 |
#### Model Architecture
The model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:
| Parameters | Hidden Size | Layers | Heads | Sequence Length |
|----------------|-------------|--------|-------|-----------------|
| 2,795,443,200 | 2560 | 32 | 32 | 4096 |
- **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).
- **Normalization**: LayerNorm ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)) with learned bias terms as opposed to RMSNorm ([Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)).
- **Tokenizer**: GPT-NeoX ([Black et al., 2022](https://arxiv.org/abs/2204.06745)).
#### Training Data
The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), RedPajama-Data ([Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data)) and The Pile ([Gao et al., 2020](https://arxiv.org/abs/2101.00027)) both without the *Books3* subset, and StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)).
> Given the large amount of web data, we recommend fine-tuning the base StableLM-3B-4E1T for your downstream tasks.
#### Training Details
Please refer to the provided YAML configuration file [`stablelm-3b-4e1t.yml`](./configs/stablelm-3b-4e1t.yml) for complete hyperparameter settings and the [technical report](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo) for further details.
#### Downstream Results
The following zero-shot evaluations are performed with the `lm-evaluation-harness` using the [lm-bench](https://github.com/Stability-AI/lm-evaluation-harness/tree/lm-bench) branch of Stability AI's fork. Full `lm-eval` JSONs can be found in the [`evals`](./evals) directory.
| Pre-Trained Model | Average | ARC<br>Challenge | ARC<br>Easy | BoolQ | HellaSwag (✱) | LAMBADA<br>OpenAI | OpenBookQA | PIQA | SciQ | Winogrande |
| ------------------------------------------------------------------------------------- |:-----------------:|:----------------:|:-----------:|:-----:|:-------------:|:-----------------:|:----------:|:-----:|:-----:|:----------:|
| meta-llama/Llama-2-13b-hf | 71.77 | 48.63 | 79.50 | 80.52 | 79.36 | 76.77 | 35.40 | 79.05 | 94.50 | 72.22 |
| huggyllama/llama-7b | 68.84 | 41.89 | 75.25 | 75.05 | 76.22 | 73.55 | 34.40 | 78.67 | 94.60 | 69.93 |
| meta-llama/Llama-2-7b-hf | 68.75 | 43.00 | 76.26 | 77.74 | 75.94 | 73.47 | 31.40 | 77.75 | 93.60 | 69.61 |
| Qwen/Qwen-7B | 67.91 | 45.39 | 67.38 | 74.56 | 88.85 (?) | 69.67 | 32.20 | 73.99 | 93.20 | 65.98 |
| tiiuae/falcon-7b | 67.83 | 40.27 | 74.41 | 73.55 | 76.35 | 74.56 | 30.60 | 79.49 | 94.00 | 67.25 |
| mosaicml/mpt-7b | 67.36 | 40.53 | 74.92 | 73.94 | 76.17 | 68.64 | 31.40 | 78.89 | 93.70 | 68.03 |
| **stabilityai/stablelm-3b-4e1t** | 66.93 | 37.80 | 72.47 | 75.63 | 73.90 | 70.64 | 31.40 | 79.22 | 94.80 | 66.54 |
| baichuan-inc/Baichuan2-7B-Base | 66.93 | 42.24 | 75.00 | 73.09 | 72.29 | 70.99 | 30.40 | 76.17 | 94.60 | 67.56 |
| stabilityai/stablelm-base-alpha-7b-v2 | 66.89 | 38.48 | 73.19 | 70.31 | 74.27 | 74.19 | 30.40 | 78.45 | 93.90 | 68.82 |
| openlm-research/open_llama_7b_v2 | 66.32 | 38.82 | 71.93 | 71.41 | 74.65 | 71.05 | 30.20 | 79.16 | 93.80 | 65.82 |
| microsoft/phi-1_5 | 65.57 | 44.45 | 76.14 | 74.53 | 62.62 | 52.75 | 37.60 | 76.33 | 93.20 | 72.53 |
| EleutherAI/gpt-neox-20B | 65.57 | 37.88 | 72.90 | 69.48 | 71.43 | 71.98 | 29.80 | 77.42 | 93.10 | 66.14 |
| togethercomputer/RedPajama-INCITE-7B-Base | 65.07 | 37.71 | 72.35 | 70.76 | 70.33 | 71.34 | 29.00 | 77.15 | 92.70 | 64.33 |
| cerebras/btlm-3b-8k-base (§) | 63.59 | 34.90 | 70.45 | 69.63 | 69.78 | 66.23 | 27.60 | 75.84 | 92.90 | 64.96 |
| EleutherAI/pythia-12b | 62.69 | 31.83 | 70.20 | 67.31 | 67.38 | 70.64 | 26.40 | 76.28 | 90.20 | 64.01 |
| openlm-research/open_llama_3b_v2 | 62.43 | 33.87 | 67.59 | 65.69 | 69.99 | 66.74 | 26.00 | 76.66 | 92.40 | 62.90 |
| EleutherAI/gpt-j-6B | 62.34 | 33.96 | 66.96 | 65.44 | 66.24 | 68.23 | 29.00 | 75.57 | 91.50 | 64.17 |
| stabilityai/stablelm-base-alpha-3b-v2 | 62.19 | 32.42 | 67.26 | 64.56 | 68.58 | 70.25 | 26.40 | 76.01 | 92.10 | 62.12 |
| facebook/opt-6.7b | 61.85 | 30.72 | 65.66 | 66.02 | 67.20 | 67.65 | 27.60 | 76.33 | 90.10 | 65.35 |
| EleutherAI/pythia-6.9b | 60.58 | 31.83 | 67.21 | 64.01 | 63.88 | 67.01 | 25.80 | 75.08 | 89.80 | 60.62 |
| EleutherAI/pythia-2.8b-deduped | 58.52 | 30.12 | 63.47 | 64.13 | 59.44 | 65.15 | 23.80 | 74.10 | 88.20 | 58.25 |
| **§** Previous 3B Pre-Trained SOTA <br>**?** Outlier Reuslts<br>**\*** Byte-length Normalized Accuracy | | | | | | | | | | |
**StableLM-3B-4E1T achieves state-of-the-art performance (September 2023) at the 3B parameter scale for open-source models** and is competitive with many of the popular contemporary 7B models, even outperforming our most recent 7B StableLM-Base-Alpha-v2.
### StableLM-Alpha v2
StableLM-Alpha v2 models significantly improve on the initial Alpha models by incorporating architectural improvements such as SwiGLU ([Shazeer, 2020](https://arxiv.org/abs/2002.05202)) and using higher-quality data sources, as discussed below. The context length for these models is 4096 tokens.
StableLM-Alpha v2 models significantly improve on the initial Alpha models by incorporating architectural improvements such as SwiGLU ([Shazeer, 2020](https://arxiv.org/abs/2002.05202)) and using higher-quality data sources, as discussed below. The context length for these models is 4096 tokens.
| Size | StableLM-Base-Alpha-v2 | Training Tokens | Parameters |
|------|----------------------------------------------------------------------------|-----------------|---------------|
@@ -34,7 +101,7 @@ StableLM-Alpha v2 models significantly improve on the initial Alpha models by in
#### Training Details
Please refer to the provided YAML configuration files for hyperparameter details. E.g. for the extended `StableLM-Alpha-3B-v2` model, see [stablelm-base-alpha-3b-v2-4k-extension.yaml](./configs/stablelm-base-alpha-3b-v2-4k-extension.yaml).
Please refer to the provided YAML configuration files for hyperparameter details. E.g. for the extended `StableLM-Alpha-3B-v2` model, see [stablelm-base-alpha-3b-v2-4k-extension.yml](./configs/stablelm-base-alpha-3b-v2-4k-extension.yml).
Following similar work, we use a multi-stage approach to context length extension ([Nijkamp et al., 2023](https://blog.salesforceairesearch.com/xgen/)), scheduling 1 trillion tokens at context length 2048 followed by 100 billion tokens at 4096. We found that sequence length warmup ([Li et al., 2022](https://arxiv.org/abs/2108.06084)) helped stabilize early spikes during the first ~80 billion tokens of pre-training. However, it was not applied to the final runs due to significant throughput penalties as length shapes grew across the curriculum.
configs/stablelm-3b-4e1t.yml
Executable
+133
View File
@@ -0,0 +1,133 @@
{
# parallelism settings
"pipe-parallel-size": 1,
"model-parallel-size": 1,
# model settings
"num-layers": 32,
"hidden-size": 2560,
"num-attention-heads": 32,
"seq-length": 4096,
"max-position-embeddings": 4096,
# architecture design
"attention_head_type": "multihead",
"norm": "layernorm",
"pos-emb": "rotary",
"rotary_pct": 0.25,
"rotary_interleaved": false,
"mlp_multiple_of": 256,
"mlp_type": "gated",
"activation": "silu",
"no-weight-tying": true,
"gpt_j_residual": false,
"gpt_j_tied": false,
"output_layer_parallelism": "column",
# init methods
"output_layer_init_method": "scaled_normal",
# biases
"use_bias_in_norms": true,
"use_bias_in_attn_linear": false,
"use_bias_in_mlp": false,
# fused ops
"attention-config": [[["flash"], 32]],
# optimizer settings
"optimizer": {
"type": "Adam",
"params": {
"lr": 3.2e-4,
"betas": [0.9, 0.95],
"eps": 1.0e-6
},
},
"min_lr": 1.28e-5, # Decay to 4% of lr
# 955_000 iters ~= 4.0T tokens at bs=4M
"train-iters": 955_000,
"lr-decay-iters": 955_000,
"lr-decay-style": "cosine",
"warmup": 0.005, # ~5k warmup steps
# ZeRO settings
"zero_optimization": {
"stage": 1,
"allgather_partitions": true,
"allgather_bucket_size": 1260000000,
"overlap_comm": true,
"reduce_scatter": true,
"reduce_bucket_size": 1260000000,
"contiguous_gradients": true,
"cpu_offload": false,
},
# batch / data settings
"train_micro_batch_size_per_gpu": 4,
"gradient_accumulation_steps": 1,
"data-impl": "mmap",
"eval-interval": 5_000,
"eval-iters": 10,
"eval_batch_size": 8,
"eval_tasks": [],
# activation checkpointing
"checkpoint-activations": true,
"checkpoint-num-layers": 1,
"partition-activations": true,
"synchronize-each-layer": true,
# regularization
"gradient_clipping": 1.0,
"weight-decay": 0.1,
"hidden-dropout": 0,
"attention-dropout": 0,
# precision settings
"bf16": { "enabled": true },
"precision": "bfloat16",
"full_precision_lm_cross_entropy": true,
# misc. training settings
"num-workers": 2,
"distributed-backend": "nccl",
# checkpoint settings
"checkpoint-factor": 2_000,
"s3_sync_interval": 10_000,
"extra-save-iters": [0],
"save": "",
"load": "",
"s3_path": "",
# data path settings
"train-data-paths": [],
"train-data-weights": [],
"valid-data-paths": [],
"valid-data-weights": [],
"test-data-paths": [],
"test-data-weights": [],
# tokenizer settings
"tokenizer-type": "HFTokenizer",
"vocab-file": "neox-tokenizer-vocab.json",
# log settings
"log-interval": 10,
"steps_per_print": 10,
"wall_clock_breakdown": true,
"use_wandb": true,
"wandb_host": "",
"wandb_team": "",
"wandb_project": "",
"wandb_group": "",
"wandb_name": "",
# "wandb_id": "",
# "wandb_resume": "",
# multi-node launcher
"launcher": "slurm",
"deepspeed_slurm": true,
}
configs/stablelm-base-alpha-3b-v2-4k-extension.yaml → configs/stablelm-base-alpha-3b-v2-4k-extension.yml
View File
configs/stablelm-base-alpha-3b-v2.yaml → configs/stablelm-base-alpha-3b-v2.yml
View File
configs/stablelm-base-alpha-3b.yaml → configs/stablelm-base-alpha-3b.yml
View File
configs/stablelm-base-alpha-7b-v2-4k-extension.yaml → configs/stablelm-base-alpha-7b-v2-4k-extension.yml
View File
configs/stablelm-base-alpha-7b-v2.yaml → configs/stablelm-base-alpha-7b-v2.yml
View File
configs/stablelm-base-alpha-7b.yaml → configs/stablelm-base-alpha-7b.yml
View File
evals/external/EleutherAI-pythia-2.8b-deduped.json
Vendored Regular → Executable
View File
evals/external/EleutherAI_gpt-j-6B.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.3395904436860068,
"acc_stderr": 0.01383903976282016,
"acc_norm": 0.3651877133105802,
"acc_norm_stderr": 0.014070265519268802
},
"arc_easy": {
"acc": 0.6696127946127947,
"acc_stderr": 0.009651430216428178,
"acc_norm": 0.6224747474747475,
"acc_norm_stderr": 0.009947227833469421
},
"boolq": {
"acc": 0.654434250764526,
"acc_stderr": 0.008317463342191592
},
"hellaswag": {
"acc": 0.49522007568213505,
"acc_stderr": 0.004989553396413105,
"acc_norm": 0.6624178450507867,
"acc_norm_stderr": 0.004719187890948069
},
"lambada_openai": {
"ppl": 4.102416000764715,
"ppl_stderr": 0.08849985162393556,
"acc": 0.6823209780710265,
"acc_stderr": 0.0064863548390796605
},
"openbookqa": {
"acc": 0.29,
"acc_stderr": 0.02031317923174518,
"acc_norm": 0.382,
"acc_norm_stderr": 0.02175082059125084
},
"piqa": {
"acc": 0.7557127312295974,
"acc_stderr": 0.010024765172284244,
"acc_norm": 0.7616974972796517,
"acc_norm_stderr": 0.009940334245876224
},
"sciq": {
"acc": 0.915,
"acc_stderr": 0.008823426366942316,
"acc_norm": 0.874,
"acc_norm_stderr": 0.010499249222408035
},
"siqa": {
"acc": 0.4109518935516888,
"acc_stderr": 0.011133193398910182,
"acc_norm": 0.4508700102354145,
"acc_norm_stderr": 0.011259319269273942
},
"winogrande": {
"acc": 0.6416732438831886,
"acc_stderr": 0.013476581172567524
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=EleutherAI/gpt-j-6B,dtype=float16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/EleutherAI_gpt-neox-20B.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.378839590443686,
"acc_stderr": 0.014175915490000319,
"acc_norm": 0.40784982935153585,
"acc_norm_stderr": 0.014361097288449696
},
"arc_easy": {
"acc": 0.7289562289562289,
"acc_stderr": 0.0091209197417606,
"acc_norm": 0.6868686868686869,
"acc_norm_stderr": 0.009516303879309528
},
"boolq": {
"acc": 0.6948012232415902,
"acc_stderr": 0.00805404814192796
},
"hellaswag": {
"acc": 0.5356502688707429,
"acc_stderr": 0.004977081808179433,
"acc_norm": 0.714299940250946,
"acc_norm_stderr": 0.004508239594503832
},
"lambada_openai": {
"ppl": 3.6403044358845733,
"ppl_stderr": 0.0747797639775495,
"acc": 0.7197748884145159,
"acc_stderr": 0.0062569681407934575
},
"openbookqa": {
"acc": 0.298,
"acc_stderr": 0.02047511809298897,
"acc_norm": 0.402,
"acc_norm_stderr": 0.021948929609938602
},
"piqa": {
"acc": 0.7742110990206746,
"acc_stderr": 0.009754980670917316,
"acc_norm": 0.7845484221980413,
"acc_norm_stderr": 0.009592463115658116
},
"sciq": {
"acc": 0.931,
"acc_stderr": 0.008018934050315145,
"acc_norm": 0.89,
"acc_norm_stderr": 0.009899393819724444
},
"siqa": {
"acc": 0.4196519959058342,
"acc_stderr": 0.011167032303390547,
"acc_norm": 0.44728761514841353,
"acc_norm_stderr": 0.011251020423273035
},
"winogrande": {
"acc": 0.6614048934490924,
"acc_stderr": 0.01330016986584241
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=EleutherAI/gpt-neox-20B,dtype=float16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/EleutherAI_pythia-12b.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.318259385665529,
"acc_stderr": 0.013611993916971451,
"acc_norm": 0.3506825938566553,
"acc_norm_stderr": 0.013944635930726083
},
"arc_easy": {
"acc": 0.702020202020202,
"acc_stderr": 0.009385046066694871,
"acc_norm": 0.6372053872053872,
"acc_norm_stderr": 0.00986593675701393
},
"boolq": {
"acc": 0.6730886850152905,
"acc_stderr": 0.008204340208838751
},
"hellaswag": {
"acc": 0.5046803425612428,
"acc_stderr": 0.004989562798280524,
"acc_norm": 0.673770165305716,
"acc_norm_stderr": 0.004678743563766643
},
"lambada_openai": {
"ppl": 3.9264187287921497,
"ppl_stderr": 0.08375179732007268,
"acc": 0.7063846303124394,
"acc_stderr": 0.006344860619678724
},
"openbookqa": {
"acc": 0.264,
"acc_stderr": 0.019732885585922108,
"acc_norm": 0.372,
"acc_norm_stderr": 0.0216371979857224
},
"piqa": {
"acc": 0.7627856365614799,
"acc_stderr": 0.009924694933586366,
"acc_norm": 0.7698585418933623,
"acc_norm_stderr": 0.009820832826839798
},
"sciq": {
"acc": 0.902,
"acc_stderr": 0.009406619184621235,
"acc_norm": 0.851,
"acc_norm_stderr": 0.011266140684632175
},
"siqa": {
"acc": 0.4201637666325486,
"acc_stderr": 0.011168911571162012,
"acc_norm": 0.44268167860798363,
"acc_norm_stderr": 0.011239482425741961
},
"winogrande": {
"acc": 0.6400947119179163,
"acc_stderr": 0.013489609590266804
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=EleutherAI/pythia-12b,dtype=float16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "1",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/EleutherAI_pythia-6.9b.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.318259385665529,
"acc_stderr": 0.013611993916971451,
"acc_norm": 0.3532423208191126,
"acc_norm_stderr": 0.013967822714840055
},
"arc_easy": {
"acc": 0.6721380471380471,
"acc_stderr": 0.009632587076170013,
"acc_norm": 0.6106902356902357,
"acc_norm_stderr": 0.010005212782878145
},
"boolq": {
"acc": 0.6400611620795107,
"acc_stderr": 0.008394940698368873
},
"hellaswag": {
"acc": 0.48048197570205137,
"acc_stderr": 0.0049859782149379184,
"acc_norm": 0.6388169687313284,
"acc_norm_stderr": 0.004793617835645056
},
"lambada_openai": {
"ppl": 4.457423605452202,
"ppl_stderr": 0.10007464956313723,
"acc": 0.670095090238696,
"acc_stderr": 0.00655050345779628
},
"openbookqa": {
"acc": 0.258,
"acc_stderr": 0.019586711785215837,
"acc_norm": 0.372,
"acc_norm_stderr": 0.0216371979857224
},
"piqa": {
"acc": 0.750816104461371,
"acc_stderr": 0.01009188277012022,
"acc_norm": 0.76550598476605,
"acc_norm_stderr": 0.009885203143240538
},
"sciq": {
"acc": 0.898,
"acc_stderr": 0.00957536880165389,
"acc_norm": 0.84,
"acc_norm_stderr": 0.011598902298689004
},
"siqa": {
"acc": 0.40736949846468784,
"acc_stderr": 0.011118216651888717,
"acc_norm": 0.42988741044012285,
"acc_norm_stderr": 0.011202283451328794
},
"winogrande": {
"acc": 0.606156274664562,
"acc_stderr": 0.013732114472668745
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=EleutherAI/pythia-6.9b,dtype=float16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/Qwen-Qwen-7B-Chat.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.45563139931740615,
"acc_stderr": 0.014553749939306863,
"acc_norm": 0.4667235494880546,
"acc_norm_stderr": 0.014578995859605797
},
"arc_easy": {
"acc": 0.6919191919191919,
"acc_stderr": 0.00947388707582633,
"acc_norm": 0.6447811447811448,
"acc_norm_stderr": 0.009820245899287122
},
"boolq": {
"acc": 0.7168195718654434,
"acc_stderr": 0.007880052012351937
},
"hellaswag": {
"acc": 0.6792471619199363,
"acc_stderr": 0.004658120152230809,
"acc_norm": 0.8497311292571201,
"acc_norm_stderr": 0.003566044777327419
},
"lambada_openai": {
"ppl": 4.258406715958618,
"ppl_stderr": 0.1263040238577051,
"acc": 0.6547642150203765,
"acc_stderr": 0.006623879809039193
},
"openbookqa": {
"acc": 0.356,
"acc_stderr": 0.021434712356072652,
"acc_norm": 0.462,
"acc_norm_stderr": 0.02231833811987053
},
"piqa": {
"acc": 0.7872687704026116,
"acc_stderr": 0.00954822312304733,
"acc_norm": 0.7992383025027203,
"acc_norm_stderr": 0.00934596167482341
},
"sciq": {
"acc": 0.907,
"acc_stderr": 0.00918887563499669,
"acc_norm": 0.806,
"acc_norm_stderr": 0.012510816141264359
},
"siqa": {
"acc": 0.4703172978505629,
"acc_stderr": 0.011294116144908552,
"acc_norm": 0.47389969293756395,
"acc_norm_stderr": 0.01129864516098083
},
"winogrande": {
"acc": 0.6858721389108129,
"acc_stderr": 0.01304541671607256
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=Qwen/Qwen-7B-Chat,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:4",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/Qwen-Qwen-7B.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.4539249146757679,
"acc_stderr": 0.014549221105171865,
"acc_norm": 0.49146757679180886,
"acc_norm_stderr": 0.014609263165632186
},
"arc_easy": {
"acc": 0.6738215488215489,
"acc_stderr": 0.009619849417035167,
"acc_norm": 0.6519360269360269,
"acc_norm_stderr": 0.009774627600259012
},
"boolq": {
"acc": 0.745565749235474,
"acc_stderr": 0.007617690099234367
},
"hellaswag": {
"acc": 0.7305317665803625,
"acc_stderr": 0.004427767996301626,
"acc_norm": 0.8884684325831508,
"acc_norm_stderr": 0.0031414591751392695
},
"lambada_openai": {
"ppl": 4.014976946118802,
"ppl_stderr": 0.08858949240464632,
"acc": 0.6966815447312246,
"acc_stderr": 0.006404402872809113
},
"openbookqa": {
"acc": 0.322,
"acc_stderr": 0.02091666833001988,
"acc_norm": 0.448,
"acc_norm_stderr": 0.022261697292270143
},
"piqa": {
"acc": 0.7399347116430903,
"acc_stderr": 0.010234893249061308,
"acc_norm": 0.749183895538629,
"acc_norm_stderr": 0.010113869547069044
},
"sciq": {
"acc": 0.932,
"acc_stderr": 0.007964887911291603,
"acc_norm": 0.908,
"acc_norm_stderr": 0.009144376393151117
},
"siqa": {
"acc": 0.49437052200614123,
"acc_stderr": 0.011313353423379522,
"acc_norm": 0.5240532241555783,
"acc_norm_stderr": 0.01130097128912773
},
"winogrande": {
"acc": 0.65982636148382,
"acc_stderr": 0.013315218762417397
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=True,pretrained=Qwen/Qwen-7B,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:4",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/baichuan-inc_Baichuan2-7B-Base.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.42235494880546076,
"acc_stderr": 0.014434138713379977,
"acc_norm": 0.431740614334471,
"acc_norm_stderr": 0.014474591427196202
},
"arc_easy": {
"acc": 0.75,
"acc_stderr": 0.008885233166386385,
"acc_norm": 0.7281144781144782,
"acc_norm_stderr": 0.0091297958673105
},
"boolq": {
"acc": 0.7308868501529052,
"acc_stderr": 0.007756844290794318
},
"hellaswag": {
"acc": 0.5366460864369648,
"acc_stderr": 0.004976361454341348,
"acc_norm": 0.7228639713204541,
"acc_norm_stderr": 0.004466695023677831
},
"lambada_openai": {
"ppl": 3.6834487322486384,
"ppl_stderr": 0.07653758845393456,
"acc": 0.7098777411216767,
"acc_stderr": 0.006322580641394925
},
"openbookqa": {
"acc": 0.304,
"acc_stderr": 0.02059164957122493,
"acc_norm": 0.394,
"acc_norm_stderr": 0.021874299301689253
},
"piqa": {
"acc": 0.7616974972796517,
"acc_stderr": 0.009940334245876209,
"acc_norm": 0.7736670293797606,
"acc_norm_stderr": 0.009763294246879418
},
"sciq": {
"acc": 0.946,
"acc_stderr": 0.007150883521295436,
"acc_norm": 0.913,
"acc_norm_stderr": 0.008916866630745916
},
"siqa": {
"acc": 0.41760491299897645,
"acc_stderr": 0.011159391894922486,
"acc_norm": 0.44779938587512796,
"acc_norm_stderr": 0.011252242102001767
},
"winogrande": {
"acc": 0.675611681136543,
"acc_stderr": 0.01315722572664164
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=baichuan-inc/Baichuan2-7B-Base,dtype=bfloat16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/bigscience-bloom-3b.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.27986348122866894,
"acc_stderr": 0.013119040897725922,
"acc_norm": 0.3037542662116041,
"acc_norm_stderr": 0.013438909184778757
},
"arc_easy": {
"acc": 0.5942760942760943,
"acc_stderr": 0.010075755540128871,
"acc_norm": 0.5328282828282829,
"acc_norm_stderr": 0.010237645778853869
},
"boolq": {
"acc": 0.617125382262997,
"acc_stderr": 0.008501734385335953
},
"hellaswag": {
"acc": 0.4137621987651862,
"acc_stderr": 0.004915003499517833,
"acc_norm": 0.545309699263095,
"acc_norm_stderr": 0.004969251445596341
},
"lambada_openai": {
"ppl": 9.094700256232823,
"ppl_stderr": 0.2652067493709512,
"acc": 0.5173685231903745,
"acc_stderr": 0.006961773596960152
},
"openbookqa": {
"acc": 0.218,
"acc_stderr": 0.01848337822317886,
"acc_norm": 0.322,
"acc_norm_stderr": 0.020916668330019882
},
"piqa": {
"acc": 0.705658324265506,
"acc_stderr": 0.010633311470347498,
"acc_norm": 0.7067464635473341,
"acc_norm_stderr": 0.010621818421101931
},
"sciq": {
"acc": 0.891,
"acc_stderr": 0.009859828407037191,
"acc_norm": 0.816,
"acc_norm_stderr": 0.012259457340938588
},
"siqa": {
"acc": 0.4017400204708291,
"acc_stderr": 0.011093444192711183,
"acc_norm": 0.4314227226202661,
"acc_norm_stderr": 0.011207148736838392
},
"winogrande": {
"acc": 0.584846093133386,
"acc_stderr": 0.01384868408665859
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=True,pretrained=bigscience/bloom-3b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": null,
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/bigscience-bloom-7b1.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.302901023890785,
"acc_stderr": 0.013428241573185349,
"acc_norm": 0.33532423208191126,
"acc_norm_stderr": 0.013796182947785564
},
"arc_easy": {
"acc": 0.6494107744107744,
"acc_stderr": 0.009791003829831557,
"acc_norm": 0.5736531986531986,
"acc_norm_stderr": 0.010147858603835144
},
"boolq": {
"acc": 0.6284403669724771,
"acc_stderr": 0.008451598145076589
},
"hellaswag": {
"acc": 0.4649472216689902,
"acc_stderr": 0.0049775044466089996,
"acc_norm": 0.6228838876717785,
"acc_norm_stderr": 0.004836738514051334
},
"lambada_openai": {
"ppl": 6.619927277080142,
"ppl_stderr": 0.1762520708430581,
"acc": 0.5755870366776635,
"acc_stderr": 0.006885918770006387
},
"openbookqa": {
"acc": 0.252,
"acc_stderr": 0.019435727282249536,
"acc_norm": 0.358,
"acc_norm_stderr": 0.021461434862859122
},
"piqa": {
"acc": 0.7274211099020674,
"acc_stderr": 0.010389256803296021,
"acc_norm": 0.7366702937976061,
"acc_norm_stderr": 0.010276185322196764
},
"sciq": {
"acc": 0.901,
"acc_stderr": 0.009449248027662765,
"acc_norm": 0.845,
"acc_norm_stderr": 0.011450157470799475
},
"siqa": {
"acc": 0.4211873080859775,
"acc_stderr": 0.011172633149198374,
"acc_norm": 0.4498464687819857,
"acc_norm_stderr": 0.011257008360485692
},
"winogrande": {
"acc": 0.6464088397790055,
"acc_stderr": 0.013436541262599948
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=True,pretrained=bigscience/bloom-7b1,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:5",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/cerebras-btlm-3b-8k-base.json
Vendored Regular → Executable
View File
evals/external/facebook-opt-2.7b.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.2687713310580205,
"acc_stderr": 0.012955065963710691,
"acc_norm": 0.31313993174061433,
"acc_norm_stderr": 0.013552671543623504
},
"arc_easy": {
"acc": 0.6077441077441077,
"acc_stderr": 0.010018744689650043,
"acc_norm": 0.5429292929292929,
"acc_norm_stderr": 0.01022189756425603
},
"boolq": {
"acc": 0.6033639143730887,
"acc_stderr": 0.008556148582032
},
"hellaswag": {
"acc": 0.4584744074885481,
"acc_stderr": 0.004972543127767873,
"acc_norm": 0.6059549890460068,
"acc_norm_stderr": 0.0048764594346198
},
"lambada_openai": {
"ppl": 5.119857738610855,
"ppl_stderr": 0.11991227443177162,
"acc": 0.6357461672811954,
"acc_stderr": 0.006704339729528894
},
"openbookqa": {
"acc": 0.25,
"acc_stderr": 0.019384310743640384,
"acc_norm": 0.352,
"acc_norm_stderr": 0.021380042385946048
},
"piqa": {
"acc": 0.7383025027203483,
"acc_stderr": 0.01025563077270823,
"acc_norm": 0.7480957562568009,
"acc_norm_stderr": 0.010128421335088685
},
"sciq": {
"acc": 0.858,
"acc_stderr": 0.011043457699378227,
"acc_norm": 0.79,
"acc_norm_stderr": 0.012886662332274536
},
"siqa": {
"acc": 0.4083930399181167,
"acc_stderr": 0.011122558066098069,
"acc_norm": 0.44268167860798363,
"acc_norm_stderr": 0.011239482425741961
},
"winogrande": {
"acc": 0.6101026045777427,
"acc_stderr": 0.013707547317008462
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=True,pretrained=facebook/opt-2.7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:2",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/facebook-opt-6.7b.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.30716723549488056,
"acc_stderr": 0.013481034054980945,
"acc_norm": 0.34812286689419797,
"acc_norm_stderr": 0.013921008595179338
},
"arc_easy": {
"acc": 0.6565656565656566,
"acc_stderr": 0.00974381736896,
"acc_norm": 0.6014309764309764,
"acc_norm_stderr": 0.010046455400477947
},
"boolq": {
"acc": 0.6602446483180429,
"acc_stderr": 0.00828377201314756
},
"hellaswag": {
"acc": 0.5051782513443537,
"acc_stderr": 0.004989513809408587,
"acc_norm": 0.6719776936865166,
"acc_norm_stderr": 0.004685334844038652
},
"lambada_openai": {
"ppl": 4.252533670173101,
"ppl_stderr": 0.0927155022353668,
"acc": 0.6764991267222977,
"acc_stderr": 0.006517535744360227
},
"openbookqa": {
"acc": 0.276,
"acc_stderr": 0.020011219298073524,
"acc_norm": 0.372,
"acc_norm_stderr": 0.0216371979857224
},
"piqa": {
"acc": 0.7633297062023939,
"acc_stderr": 0.009916841655042804,
"acc_norm": 0.766050054406964,
"acc_norm_stderr": 0.009877236895137437
},
"sciq": {
"acc": 0.901,
"acc_stderr": 0.009449248027662761,
"acc_norm": 0.853,
"acc_norm_stderr": 0.011203415395160333
},
"siqa": {
"acc": 0.4263050153531218,
"acc_stderr": 0.011190503463264742,
"acc_norm": 0.4595701125895599,
"acc_norm_stderr": 0.011277022486079959
},
"winogrande": {
"acc": 0.6535122336227308,
"acc_stderr": 0.013373773411685644
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=True,pretrained=facebook/opt-6.7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:2",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/huggyllama-llama-7b.json
Vendored Executable
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.4189419795221843,
"acc_stderr": 0.014418106953639013,
"acc_norm": 0.4462457337883959,
"acc_norm_stderr": 0.014526705548539982
},
"arc_easy": {
"acc": 0.7525252525252525,
"acc_stderr": 0.00885511441483471,
"acc_norm": 0.7285353535353535,
"acc_norm_stderr": 0.009125362970360627
},
"boolq": {
"acc": 0.7504587155963303,
"acc_stderr": 0.0075688020241860285
},
"hellaswag": {
"acc": 0.5696076478789086,
"acc_stderr": 0.004941191607317913,
"acc_norm": 0.7621987651862179,
"acc_norm_stderr": 0.004248666961833349
},
"lambada_openai": {
"ppl": 3.4882593690160553,
"ppl_stderr": 0.068517853205977,
"acc": 0.7354938870560839,
"acc_stderr": 0.006144965702579053
},
"openbookqa": {
"acc": 0.344,
"acc_stderr": 0.02126575803797874,
"acc_norm": 0.444,
"acc_norm_stderr": 0.02224224437573102
},
"piqa": {
"acc": 0.7867247007616975,
"acc_stderr": 0.00955712122586133,
"acc_norm": 0.7916213275299239,
"acc_norm_stderr": 0.009476125383049459
},
"sciq": {
"acc": 0.946,
"acc_stderr": 0.007150883521295433,
"acc_norm": 0.928,
"acc_norm_stderr": 0.008178195576218681
},
"siqa": {
"acc": 0.4483111566018424,
"acc_stderr": 0.01125345173122258,
"acc_norm": 0.4703172978505629,
"acc_norm_stderr": 0.011294116144908554
},
"winogrande": {
"acc": 0.6992896606156275,
"acc_stderr": 0.012888010494704723
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=False,pretrained=huggyllama/llama-7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:2",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/kittn_mistral-7B-v0.1-hf.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.5025597269624573,
"acc_stderr": 0.014611199329843777,
"acc_norm": 0.5435153583617748,
"acc_norm_stderr": 0.014555949760496439
},
"arc_easy": {
"acc": 0.8080808080808081,
"acc_stderr": 0.00808080808080796,
"acc_norm": 0.7954545454545454,
"acc_norm_stderr": 0.008276958800002997
},
"boolq": {
"acc": 0.8363914373088684,
"acc_stderr": 0.006469941343840763
},
"hellaswag": {
"acc": 0.6127265484963155,
"acc_stderr": 0.0048613146132868434,
"acc_norm": 0.8105954989046007,
"acc_norm_stderr": 0.003910288117015163
},
"lambada_openai": {
"ppl": 3.181388367482107,
"ppl_stderr": 0.058369197945258765,
"acc": 0.7568406753347564,
"acc_stderr": 0.0059766767751295085
},
"openbookqa": {
"acc": 0.328,
"acc_stderr": 0.021017027165175485,
"acc_norm": 0.44,
"acc_norm_stderr": 0.022221331534143057
},
"piqa": {
"acc": 0.8079434167573449,
"acc_stderr": 0.00919074029512649,
"acc_norm": 0.8215451577801959,
"acc_norm_stderr": 0.008933575463062072
},
"sciq": {
"acc": 0.959,
"acc_stderr": 0.006273624021118748,
"acc_norm": 0.938,
"acc_norm_stderr": 0.0076298239962803065
},
"siqa": {
"acc": 0.4273285568065507,
"acc_stderr": 0.01119393034055127,
"acc_norm": 0.4570112589559877,
"acc_norm_stderr": 0.01127217546233142
},
"winogrande": {
"acc": 0.7403314917127072,
"acc_stderr": 0.012322700705552673
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=kittn/mistral-7B-v0.1-hf,dtype=bfloat16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=False",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/meta-llama-Llama-2-13b-hf.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.4863481228668942,
"acc_stderr": 0.014605943429860945,
"acc_norm": 0.492320819112628,
"acc_norm_stderr": 0.01460966744089257
},
"arc_easy": {
"acc": 0.79503367003367,
"acc_stderr": 0.008283277600626398,
"acc_norm": 0.7760942760942761,
"acc_norm_stderr": 0.008553779114531757
},
"boolq": {
"acc": 0.8051987767584098,
"acc_stderr": 0.006926916185348359
},
"hellaswag": {
"acc": 0.6011750647281418,
"acc_stderr": 0.004886559008754982,
"acc_norm": 0.7935670185222067,
"acc_norm_stderr": 0.004039176806180289
},
"lambada_openai": {
"ppl": 3.0445239988325836,
"ppl_stderr": 0.05613639700817276,
"acc": 0.767708131185717,
"acc_stderr": 0.005883383348944357
},
"openbookqa": {
"acc": 0.354,
"acc_stderr": 0.021407582047916447,
"acc_norm": 0.454,
"acc_norm_stderr": 0.02228814759117695
},
"piqa": {
"acc": 0.7905331882480957,
"acc_stderr": 0.0094943029798198,
"acc_norm": 0.8063112078346029,
"acc_norm_stderr": 0.009220384152336643
},
"sciq": {
"acc": 0.945,
"acc_stderr": 0.007212976294639239,
"acc_norm": 0.933,
"acc_norm_stderr": 0.007910345983177549
},
"siqa": {
"acc": 0.4278403275332651,
"acc_stderr": 0.01119562541819821,
"acc_norm": 0.4503582395087001,
"acc_norm_stderr": 0.01125816983012229
},
"winogrande": {
"acc": 0.7221783741120757,
"acc_stderr": 0.012588918183871603
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=False,pretrained=meta-llama/Llama-2-13b-hf,trust_remote_code=True,low_cpu_mem_usage=True,dtype=bfloat16",
"num_fewshot": 0,
"batch_size": "16",
"batch_sizes": [],
"device": "cuda:5",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/meta-llama-Llama-2-7b.json
Vendored Regular → Executable
View File
evals/external/microsoft-phi-1_5.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.4445392491467577,
"acc_stderr": 0.014521226405627079,
"acc_norm": 0.4803754266211604,
"acc_norm_stderr": 0.014600132075947092
},
"arc_easy": {
"acc": 0.7613636363636364,
"acc_stderr": 0.008746465140706127,
"acc_norm": 0.7314814814814815,
"acc_norm_stderr": 0.009094042554994847
},
"boolq": {
"acc": 0.7452599388379205,
"acc_stderr": 0.007620703281690057
},
"hellaswag": {
"acc": 0.47988448516231824,
"acc_stderr": 0.004985741706385726,
"acc_norm": 0.6261700856403107,
"acc_norm_stderr": 0.004828305041904399
},
"lambada_openai": {
"ppl": 8.945202930351474,
"ppl_stderr": 0.3005661666926477,
"acc": 0.527459732194838,
"acc_stderr": 0.0069554645156210786
},
"openbookqa": {
"acc": 0.376,
"acc_stderr": 0.021683827539286115,
"acc_norm": 0.482,
"acc_norm_stderr": 0.02236856511738799
},
"piqa": {
"acc": 0.7633297062023939,
"acc_stderr": 0.009916841655042806,
"acc_norm": 0.7578890097932536,
"acc_norm_stderr": 0.009994371269104381
},
"sciq": {
"acc": 0.932,
"acc_stderr": 0.007964887911291603,
"acc_norm": 0.916,
"acc_norm_stderr": 0.00877616208949113
},
"siqa": {
"acc": 0.5537359263050153,
"acc_stderr": 0.01124854090154796,
"acc_norm": 0.5957011258955988,
"acc_norm_stderr": 0.0111048923983008
},
"winogrande": {
"acc": 0.7253354380426204,
"acc_stderr": 0.012544516005117187
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=False,pretrained=microsoft/phi-1_5,trust_remote_code=True,low_cpu_mem_usage=True,dtype=bfloat16",
"num_fewshot": 0,
"batch_size": "16",
"batch_sizes": [],
"device": "cuda:5",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/mosaicml-mpt-7b.json
Vendored Regular → Executable
View File
evals/external/openlm-research-open_llama_13b
Vendored Executable
+32
View File
@@ -0,0 +1,32 @@
{
"results": {
"lambada_openai": {
"ppl": 3.569586285393247,
"ppl_stderr": 0.07265272740503036,
"acc": 0.7209392586842616,
"acc_stderr": 0.006249003708978234
},
"sciq": {
"acc": 0.941,
"acc_stderr": 0.007454835650406722,
"acc_norm": 0.914,
"acc_norm_stderr": 0.008870325962594766
}
},
"versions": {
"lambada_openai": 0,
"sciq": 0
},
"config": {
"model": "gpt2",
"model_args": "use_fast=False,pretrained=openlm-research/open_llama_13b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "2",
"batch_sizes": [],
"device": "cuda:3",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/openlm-research-open_llama_3b_v2.json
Vendored Regular → Executable
View File
evals/external/openlm-research-open_llama_7b_v2.json
Vendored Regular → Executable
View File
evals/external/tiiuae_falcon-7b.json
Vendored
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.40273037542662116,
"acc_stderr": 0.014332236306790149,
"acc_norm": 0.43686006825938567,
"acc_norm_stderr": 0.014494421584256519
},
"arc_easy": {
"acc": 0.7441077441077442,
"acc_stderr": 0.008953950243013991,
"acc_norm": 0.7079124579124579,
"acc_norm_stderr": 0.00933070561656907
},
"boolq": {
"acc": 0.735474006116208,
"acc_stderr": 0.007714546144910642
},
"hellaswag": {
"acc": 0.5771758613821948,
"acc_stderr": 0.004929983692795067,
"acc_norm": 0.7634933280223063,
"acc_norm_stderr": 0.004240683281093403
},
"lambada_openai": {
"ppl": 3.3698354504387775,
"ppl_stderr": 0.06490807599596318,
"acc": 0.7455850960605472,
"acc_stderr": 0.006067809764031527
},
"openbookqa": {
"acc": 0.306,
"acc_stderr": 0.020629569998345403,
"acc_norm": 0.442,
"acc_norm_stderr": 0.02223197069632112
},
"piqa": {
"acc": 0.794885745375408,
"acc_stderr": 0.009420971671017915,
"acc_norm": 0.8057671381936888,
"acc_norm_stderr": 0.009230209366168272
},
"sciq": {
"acc": 0.94,
"acc_stderr": 0.00751375115747492,
"acc_norm": 0.915,
"acc_norm_stderr": 0.008823426366942328
},
"siqa": {
"acc": 0.42067553735926305,
"acc_stderr": 0.011170778517705619,
"acc_norm": 0.4600818833162743,
"acc_norm_stderr": 0.011277955967920396
},
"winogrande": {
"acc": 0.6724546172059984,
"acc_stderr": 0.013190169546797016
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=tiiuae/falcon-7b,dtype=bfloat16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/external/togethercomputer-RedPajama-INCITE-7B-Base2.json
Vendored Regular → Executable
View File
evals/external/truthfulqa-stablelm-base-alpha-7b
Vendored
-25
View File
@@ -1,25 +0,0 @@
{
"results": {
"truthfulqa_mc": {
"mc1": 0.23745410036719705,
"mc1_stderr": 0.014896277441041836,
"mc2": 0.3995908363542637,
"mc2_stderr": 0.014371652685680641
}
},
"versions": {
"truthfulqa_mc": 1
},
"config": {
"model": "gpt2",
"model_args": "use_fast=True,pretrained=stabilityai/stablelm-base-alpha-7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda:3",
"no_cache": false,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}
evals/stablelm-3b-4e1t.json
+84
View File
@@ -0,0 +1,84 @@
{
"results": {
"arc_challenge": {
"acc": 0.3779863481228669,
"acc_stderr": 0.014169664520303103,
"acc_norm": 0.40017064846416384,
"acc_norm_stderr": 0.014317197787809169
},
"arc_easy": {
"acc": 0.7247474747474747,
"acc_stderr": 0.009164888895174743,
"acc_norm": 0.6771885521885522,
"acc_norm_stderr": 0.009593950220366743
},
"boolq": {
"acc": 0.7562691131498471,
"acc_stderr": 0.007509067459407977
},
"hellaswag": {
"acc": 0.5482971519617607,
"acc_stderr": 0.004966448380104203,
"acc_norm": 0.7389962158932484,
"acc_norm_stderr": 0.004382844128643414
},
"lambada_openai": {
"ppl": 3.827229437157901,
"ppl_stderr": 0.07919367133146168,
"acc": 0.7063846303124394,
"acc_stderr": 0.006344860619678723
},
"openbookqa": {
"acc": 0.314,
"acc_stderr": 0.020776701920308997,
"acc_norm": 0.398,
"acc_norm_stderr": 0.02191237788577997
},
"piqa": {
"acc": 0.7921653971708379,
"acc_stderr": 0.009466997964536423,
"acc_norm": 0.7976060935799782,
"acc_norm_stderr": 0.00937428968280767
},
"sciq": {
"acc": 0.948,
"acc_stderr": 0.007024624213817142,
"acc_norm": 0.919,
"acc_norm_stderr": 0.00863212103213998
},
"siqa": {
"acc": 0.41914022517911975,
"acc_stderr": 0.011165140708170328,
"acc_norm": 0.44319344933469806,
"acc_norm_stderr": 0.011240812731564952
},
"winogrande": {
"acc": 0.665351223362273,
"acc_stderr": 0.013261823629558375
}
},
"versions": {
"arc_challenge": 0,
"arc_easy": 0,
"boolq": 1,
"hellaswag": 0,
"lambada_openai": 0,
"openbookqa": 0,
"piqa": 0,
"sciq": 0,
"siqa": 0,
"winogrande": 0
},
"config": {
"model": "gpt2",
"model_args": "pretrained=stablelm-3b-4e1t,dtype=bfloat16,trust_remote_code=True,low_cpu_mem_usage=True,use_fast=True",
"num_fewshot": 0,
"batch_size": "8",
"batch_sizes": [],
"device": "cuda",
"no_cache": true,
"limit": null,
"bootstrap_iters": 100000,
"description_dict": {}
}
}