release: add version 2 alpha models (#95)
This commit is contained in:
Jonathan Tow
@@ -7,18 +7,79 @@ This repository contains Stability AI's ongoing development of the StableLM seri
|
||||
|
||||
## News
|
||||
|
||||
*August 5, 2023*
|
||||
|
||||
- Released patched StableLM-Alpha v2 models with 3B and 7B parameters.
|
||||
|
||||
*April 28, 2023*
|
||||
|
||||
- Released StableVicuna-13B, our RLHF fine-tune of [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), which itself is a fine-tune of [LLaMA-13B](https://github.com/facebookresearch/llama). Delta weights over the original Llama model is released under ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)).
|
||||
|
||||
*April 20, 2023*
|
||||
|
||||
- Released initial set of StableLM-alpha models, with 3B and 7B parameters. 15B and 30B models are on the way. Base models are released under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).
|
||||
- Released initial set of StableLM-Alpha models, with 3B and 7B parameters. Base models are released under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).
|
||||
|
||||
- Try to chat with our 7B model, `StableLM-Tuned-Alpha-7B`, on [Hugging Face Spaces](https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat).
|
||||
|
||||
## Models
|
||||
|
||||
### StableLM-Alpha v2
|
||||
|
||||
StableLM-Alpha v2 models significantly improve on the initial Alpha models by incorporating architectural improvements such as SwiGLU ([Shazeer, 2020](https://arxiv.org/abs/2002.05202)) and using higher-quality data sources, as discussed below. The context length for these models is 4096 tokens.
|
||||
|
||||
| Size | StableLM-Base-Alpha-v2 | Training Tokens | Parameters |
|
||||
|------|----------------------------------------------------------------------------|-----------------|---------------|
|
||||
| 3B | [checkpoint](https://huggingface.co/stabilityai/stablelm-base-alpha-3b-v2) | 1.1T | 2,796,431,360 |
|
||||
| 7B | [checkpoint](https://huggingface.co/stabilityai/stablelm-base-alpha-7b-v2) | 1.1T | 6,890,209,280 |
|
||||
|
||||
#### Training Details
|
||||
|
||||
Please refer to the provided YAML configuration files for hyperparameter details. E.g. for the extended `StableLM-Alpha-3B-v2` model, see [stablelm-base-alpha-3b-v2-4k-extension.yaml](./configs/stablelm-base-alpha-3b-v2-4k-extension.yaml).
|
||||
|
||||
Following similar work, we use a multi-stage approach to context length extension ([Nijkamp et al., 2023](https://blog.salesforceairesearch.com/xgen/)), scheduling 1 trillion tokens at context length 2048 followed by 100 billion tokens at 4096. We found that sequence length warmup ([Li et al., 2022](https://arxiv.org/abs/2108.06084)) helped stabilize early spikes during the first ~80 billion tokens of pre-training. However, it was not applied to the final runs due to significant throughput penalties as length shapes grew across the curriculum.
|
||||
|
||||
#### Training Data
|
||||
|
||||
The most impactful changes for StableLM-Alpha-v2 downstream performance were in the usage of higher quality data sources and mixtures; specifically, the use of [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) and [C4](https://huggingface.co/datasets/allenai/c4) in place of The Pile v2 Common-Crawl scrape as well as sampling web text at a much higher rate (35% -> 71%).
|
||||
|
||||
The first pre-training stage relies on 1 trillion tokens sourced from a mix of the public Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), RedPajama-Data ([Together Computer., 2023](https://github.com/togethercomputer/RedPajama-Data)), The Pile ([Gao et al., 2020](https://arxiv.org/abs/2101.00027)), and internal datasets with web text sampled at a rate of 71%.
|
||||
|
||||
In the second stage, we include the StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)) dataset and down sample web text to 55% while increasing sampling proportions of naturally long text examples in the aforementioned sources.
|
||||
|
||||
#### Evaluation
|
||||
|
||||
The following zero-shot evaluations are performed with the `lm-evaluation-harness` at commit [`df3da98c5405deafd519c2ddca52bb7c3fe36bef`](https://github.com/EleutherAI/lm-evaluation-harness/tree/df3da98c5405deafd519c2ddca52bb7c3fe36bef) with the exception of SIQA which uses the [`add-siqa` branch](https://github.com/EleutherAI/lm-evaluation-harness/tree/add-siqa) with prompt format
|
||||
`{doc['context']}\nQuestion: {doc['question']}\nAnswer:`.
|
||||
|
||||
| Model | ARC Challenge✱ | ARC Easy✱ | BoolQ | HellaSwag✱ | LAMBADA<br>OpenAI | OpenBookQA | PIQA | SIQA | TruthfulQA▲ | Winogrande | Average |
|
||||
| ------------------------- |:---------------:|:----------:|:-----:|:-----------:|:-----------------:|:----------:|:-----:|:-----:|:------------:|:----------:|:-------:|
|
||||
| **StableLM-Alpha-7B-v2** | 40.53 | 69.11 | 70.31 | 74.27 | 74.19 | 30.40 | 78.45 | 42.43 | 36.46 | 68.82 | 58.50 |
|
||||
| **StableLM-Alpha-3B-v2** | 35.07 | 63.26 | 64.56 | 68.58 | 70.25 | 26.40 | 76.01 | 42.48 | 35.87 | 62.12 | 54.46 |
|
||||
| LLaMA-2-7B | 46.16 | 74.54 | 77.74 | 75.94 | 73.47 | 31.40 | 77.75 | 43.50 | 38.97 | 69.61 | 60.91 |
|
||||
| MTP-7B | 41.89 | 70.03 | 73.94 | 76.17 | 68.64 | 31.40 | 78.89 | 45.14 | 33.49 | 68.03 | 58.76 |
|
||||
| OpenLLaMA-7B-v2 | 42.41 | 69.65 | 71.41 | 74.65 | 71.05 | 30.20 | 79.16 | 41.97 | 34.57 | 65.82 | 58.09 |
|
||||
| RedPajama-INCITE-7B-Base | 39.42 | 69.19 | 70.76 | 70.33 | 71.34 | 29.00 | 77.15 | 42.58 | 33.01 | 64.33 | 56.71 |
|
||||
| BTLM-3B-8K | 37.63 | 67.09 | 69.63 | 69.78 | 66.23 | 27.60 | 75.84 | 42.78 | 36.00 | 64.96 | 55.75 |
|
||||
| OpenLLaMA-3B-v2 | 36.09 | 63.51 | 65.69 | 69.99 | 66.74 | 26.00 | 76.66 | 41.20 | 34.59 | 62.90 | 54.34 |
|
||||
| Pythia-2.8B (deduped) | 32.94 | 59.09 | 64.13 | 59.44 | 65.15 | 23.80 | 74.10 | 40.94 | 35.56 | 58.25 | 51.34 |
|
||||
| StableLM-Alpha-7B | 27.05 | 44.87 | 60.06 | 41.22 | 55.11 | 21.40 | 66.76 | 39.46 | 39.96 | 50.12 | 44.60 |
|
||||
| StableLM-Alpha-3B | 25.77 | 42.05 | 57.65 | 38.31 | 41.72 | 17.00 | 63.82 | 35.62 | 40.53 | 52.64 | 41.51 |
|
||||
|
||||
✱: Denotes byte-length normalized accuracy (`acc_norm`) as described in [Gao, 2021](https://blog.eleuther.ai/multiple-choice-normalization/).
|
||||
|
||||
▲: We score TruthfulQA using the normalized total probability assigned to the set of true answers (`mc2`).
|
||||
|
||||
### StableLM-Alpha
|
||||
|
||||
StableLM-Alpha models are trained on a new dataset that builds on [The Pile](https://pile.eleuther.ai/), which contains 1.5 trillion tokens, roughly 3x the size of The Pile. The context length for these models is 4096 tokens.
|
||||
|
||||
As a proof-of-concept, we also fine-tuned the model with [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)'s procedure using a combination of five recent datasets for conversational agents: Stanford's [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), Nomic-AI's [gpt4all](https://github.com/nomic-ai/gpt4all), RyokoAI's [ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) datasets, Databricks labs' [Dolly](https://github.com/databrickslabs/dolly), and Anthropic's [HH](https://github.com/anthropics/hh-rlhf). We will be releasing these models as StableLM-Tuned-Alpha.
|
||||
|
||||
| Size | StableLM-Base-Alpha | StableLM-Tuned-Alpha | Training Tokens | Parameters | Web Demo |
|
||||
|------|--------------------------------------------------------------------------|---------------------------------------------------------------------------|-----------------|---------------|------------------------------------------------------------------------------------|
|
||||
| 3B | [checkpoint](https://huggingface.co/stabilityai/stablelm-base-alpha-3b/) | [checkpoint](https://huggingface.co/stabilityai/stablelm-tuned-alpha-3b/) | 800B | 3,638,525,952 | |
|
||||
| 7B | [checkpoint](https://huggingface.co/stabilityai/stablelm-base-alpha-7b) | [checkpoint](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b) | 800B | 7,869,358,080 | [Hugging Face](https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat) |
|
||||
|
||||
### StableVicuna
|
||||
|
||||
StableVicuna is an RLHF fine-tune of [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), which itself is a fine-tune of [LLaMA-13B](https://github.com/facebookresearch/llama). It is our attempt at creating an open-source RLHF LLM Chatbot. This model is developed by StabilityAI's CarperAI team, with [Duy V. Phung](https://github.com/PhungVanDuy) leading the training effort.
|
||||
@@ -31,22 +92,6 @@ Please visit HuggingFace checkpoint for more information about how to combine ou
|
||||
| ---------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------- |------|
|
||||
| StableVicuna-13B | [checkpoint](https://huggingface.co/CarperAI/stable-vicuna-13b-delta/) | [Hugging Face](https://huggingface.co/spaces/CarperAI/StableVicuna/) | [](https://doi.org/10.57967/hf/0588) |
|
||||
|
||||
### StableLM-Alpha
|
||||
StableLM-Alpha models are trained on the new dataset that build on [The Pile](https://pile.eleuther.ai/), which contains 1.5 trillion tokens, roughly 3x the size of The Pile. These models will be trained on up to 1.5 trillion tokens. The context length for these models is 4096 tokens.
|
||||
|
||||
An upcoming technical report will document the model specifications and the training settings.
|
||||
|
||||
As a proof-of-concept, we also fine-tuned the model with [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)'s procedure using a combination of five recent datasets for conversational agents: Stanford's [Alpaca](https://github.com/tatsu-lab/stanford_alpaca), Nomic-AI's [gpt4all](https://github.com/nomic-ai/gpt4all), RyokoAI's [ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) datasets, Databricks labs' [Dolly](https://github.com/databrickslabs/dolly), and Anthropic's [HH](https://github.com/anthropics/hh-rlhf). We will be releasing these models as StableLM-Tuned-Alpha.
|
||||
|
||||
| Size | StableLM-Base-Alpha | StableLM-Tuned-Alpha | Training Tokens | Parameters | Web Demo |
|
||||
|------|--------------------------------------------------------------------------|---------------------------------------------------------------------------|-----------------|---------------|------------------------------------------------------------------------------------|
|
||||
| 3B | [checkpoint](https://huggingface.co/stabilityai/stablelm-base-alpha-3b/) | [checkpoint](https://huggingface.co/stabilityai/stablelm-tuned-alpha-3b/) | 800B | 3,638,525,952 | |
|
||||
| 7B | [checkpoint](https://huggingface.co/stabilityai/stablelm-base-alpha-7b) | [checkpoint](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b) | 800B | 7,869,358,080 | [Hugging Face](https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat) |
|
||||
| 15B | (in progress) | (pending) | | | |
|
||||
| 30B | (in progress) | (pending) | | | |
|
||||
| 65B | (in progress) | (pending) | | | |
|
||||
| 175B | (planned) | | | | |
|
||||
|
||||
## Quickstart
|
||||
|
||||
All StableLM models are hosted on [the Hugging Face hub](https://huggingface.co/StabilityAI). Check out this [notebook](https://github.com/Stability-AI/StableLM/blob/main/notebooks/stablelm-alpha.ipynb) to run inference with limited GPU capabilities.
|
||||
@@ -97,11 +142,11 @@ The system prompt is
|
||||
```
|
||||
|
||||
## Fun with StableLM-Tuned-Alpha
|
||||
This section contains a collection of fun cherry-picked examples of what you can do with `stablelm-tuned-alpha`.
|
||||
This section contains a collection of fun cherry-picked examples of what you can do with `stablelm-tuned-alpha`.
|
||||
|
||||
<details>
|
||||
<summary>Chit-Chat</summary>
|
||||
|
||||
|
||||
```
|
||||
User: Hey! My name is John nice to meet you!
|
||||
|
||||
@@ -123,12 +168,12 @@ User: Ah! Right. I can order a bouquet for her online and get it delivered! Than
|
||||
|
||||
StableLM: You’re very welcome! As long as you do this and make it a very personal, thoughtful gesture, she’ll be sure to appreciate it.
|
||||
```
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Formal Writing</summary>
|
||||
|
||||
|
||||
```
|
||||
User: What would you say to a friend who is graduating high school?
|
||||
|
||||
@@ -223,7 +268,7 @@ He now knows the ins and outs of the TensorFlow programming language, and knows
|
||||
|
||||
```
|
||||
|
||||
### Humor
|
||||
### Humor
|
||||
|
||||
```
|
||||
User: What do you know about the League of Legends champion Zed? Write a short bio
|
||||
@@ -240,7 +285,7 @@ Zed has been a part of many memorable games, and his success in these games help
|
||||
|
||||
User: Write a funny conversation after a team fight between Zed and Teemo
|
||||
|
||||
StableLM:
|
||||
StableLM:
|
||||
Teemo: “Hey Zed, great game today. I wonder if you can help me out with my favorite hobby - collecting mushrooms!”
|
||||
Zed: “Teemo, please. I’m not a mushroom collector.”
|
||||
Teemo: “But of course, you are. You have the best taste in the game.”
|
||||
@@ -280,11 +325,11 @@ fib_sequence(10)
|
||||
</details>
|
||||
|
||||
## Selected failure modes of StableLM-Tuned-Alpha
|
||||
This section contains a collection of selected failure modes of `stablelm-tuned-alpha`.
|
||||
This section contains a collection of selected failure modes of `stablelm-tuned-alpha`.
|
||||
|
||||
<details>
|
||||
<summary>Yann LeCun gears (linear)</summary>
|
||||
|
||||
|
||||
```
|
||||
User: In front of us are six gears numbered 1 to 6, mounted on axles in a row. Each gear is engaged with the next gear. If gear number 3 is rotated clockwise, in which direction will gears 1 and 6 rotate?
|
||||
|
||||
@@ -292,12 +337,12 @@ StableLM: Each gear is engaged with the next gear after every other gear, so tha
|
||||
|
||||
To determine the direction in which gears 1 and 6 will rotate, we need to look at the direction of the input force. Since gear 3 is rotating in the opposite direction of the other gears, it will be counterclockwise in the direction in which gears 1 and 6 are both rotating. Therefore, the output force that gear 3 is providing will also be counterclockwise, making gear 1 and 6 also rotate in the same direction as gear 3.
|
||||
```
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Yann LeCun gears (circular)</summary>
|
||||
|
||||
|
||||
```
|
||||
User: 7 axles are equally spaced around a circle. A gear is placed on each axle such that each gear is engaged with the gear to its left and the gear to its right. The gears are numbered 1 to 7 around the circle. If gear 3 were rotated clockwise, in which direction would gear 7 rotate?
|
||||
|
||||
@@ -309,7 +354,7 @@ According to the problem, gear 3 is situated at (d/2, 2 degrees) and the gears a
|
||||
|
||||
Therefore, the answer is d = (7/2) * 2 = 5 degrees in total.
|
||||
```
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
## Request for Help
|
||||
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
{
|
||||
# parallelism settings
|
||||
"pipe-parallel-size": 1,
|
||||
"model-parallel-size": 2,
|
||||
|
||||
# model settings
|
||||
"num-layers": 32,
|
||||
"hidden-size": 2560,
|
||||
"num-attention-heads": 32,
|
||||
"seq-length": 4096,
|
||||
"max-position-embeddings": 4096,
|
||||
|
||||
# architecture design
|
||||
"attention_head_type": "multihead",
|
||||
"norm": "layernorm",
|
||||
"pos-emb": "rotary",
|
||||
"rotary_pct": 0.25,
|
||||
"rotary_interleaved": false, # GPT-NeoX style
|
||||
# NOTE: Linear Position Scaling degrades sample quality after 10B tokens - do not use yet.
|
||||
# "rotary_scaling_factor": 2, # 2048 -> 4096
|
||||
"mlp_multiple_of": 256,
|
||||
"mlp_type": "gated",
|
||||
"activation": "silu",
|
||||
"no-weight-tying": true,
|
||||
"gpt_j_residual": true,
|
||||
"gpt_j_tied": true,
|
||||
"output_layer_parallelism": "column",
|
||||
|
||||
# biases
|
||||
"use_bias_in_norms": true,
|
||||
"use_bias_in_attn_linear": false,
|
||||
"use_bias_in_mlp": false,
|
||||
|
||||
# fused ops
|
||||
"bias-gelu-fusion": false,
|
||||
"scaled-upper-triang-masked-softmax-fusion": true,
|
||||
"attention-config": [[["flash"], 32]],
|
||||
|
||||
# optimizer settings
|
||||
"optimizer": {
|
||||
"type": "Adam",
|
||||
"params": {
|
||||
"lr": 2.8e-5,
|
||||
"betas": [0.9, 0.95],
|
||||
"eps": 1.0e-6
|
||||
},
|
||||
},
|
||||
"min_lr": 2.8e-6,
|
||||
"train-iters": 50_000,
|
||||
"lr-decay-iters": 50_000,
|
||||
"lr-decay-style": "cosine",
|
||||
"warmup": 0.00,
|
||||
|
||||
# for all zero_optimization options, see https://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training
|
||||
"zero_optimization": {
|
||||
"stage": 1,
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 1260000000,
|
||||
"overlap_comm": true,
|
||||
"reduce_scatter": true,
|
||||
"reduce_bucket_size": 1260000000,
|
||||
"contiguous_gradients": true,
|
||||
"cpu_offload": false,
|
||||
},
|
||||
|
||||
# batch / data settings
|
||||
"train_micro_batch_size_per_gpu": 4,
|
||||
"gradient_accumulation_steps": 1,
|
||||
"data-impl": "mmap",
|
||||
"eval-interval": 1_000,
|
||||
"eval-iters": 10,
|
||||
"eval_batch_size": 8,
|
||||
"eval_tasks": ["lambada_openai", "piqa"],
|
||||
|
||||
# activation checkpointing
|
||||
"checkpoint-activations": true,
|
||||
"checkpoint-num-layers": 1,
|
||||
"partition-activations": true,
|
||||
"synchronize-each-layer": true,
|
||||
|
||||
# regularization
|
||||
"gradient_clipping": 1.0,
|
||||
"weight-decay": 0.0001,
|
||||
"hidden-dropout": 0,
|
||||
"attention-dropout": 0,
|
||||
|
||||
# precision settings
|
||||
"fp16": {
|
||||
"fp16": true,
|
||||
"enabled": true,
|
||||
"loss_scale": 0,
|
||||
"loss_scale_window": 1000,
|
||||
"initial_scale_power": 12,
|
||||
"hysteresis": 2,
|
||||
"min_loss_scale": 1e-10
|
||||
},
|
||||
"full_precision_lm_cross_entropy": true,
|
||||
|
||||
# misc. training settings
|
||||
"num-workers": 1,
|
||||
"distributed-backend": "nccl",
|
||||
|
||||
# checkpoint settings
|
||||
"checkpoint-factor": 2_000,
|
||||
"save": "",
|
||||
"load": "",
|
||||
"s3_path": "",
|
||||
"iteration": 245_000,
|
||||
"finetune": true,
|
||||
"no_checkpoint_arg_validation": true,
|
||||
"override_lr_scheduler": true,
|
||||
|
||||
# data path settings
|
||||
"train-data-paths": [""],
|
||||
"train-data-weights": [1.0],
|
||||
"valid-data-paths": [""],
|
||||
"valid-data-weights": [1.0],
|
||||
"test-data-paths": [""],
|
||||
"test-data-weights": [1.0],
|
||||
|
||||
# tokenizer settings
|
||||
"tokenizer-type": "HFTokenizer",
|
||||
"vocab-file": "neox-tokenizer-vocab.json",
|
||||
|
||||
# log settings
|
||||
"log-interval": 10,
|
||||
"steps_per_print": 10,
|
||||
"wall_clock_breakdown": true,
|
||||
|
||||
"use_wandb": true,
|
||||
"wandb_host": "",
|
||||
"wandb_team": "",
|
||||
"wandb_project": "",
|
||||
"wandb_group": "7B",
|
||||
"wandb_name": "stablelm-base-alpha-7b-v2-4k-finetune",
|
||||
# "wandb_id": "",
|
||||
# "wandb_resume": "must",
|
||||
|
||||
# multi-node launcher
|
||||
"launcher": "slurm",
|
||||
"deepspeed_slurm": true,
|
||||
}
|
||||
Executable
+141
@@ -0,0 +1,141 @@
|
||||
{
|
||||
# parallelism settings
|
||||
"pipe-parallel-size": 1,
|
||||
"model-parallel-size": 2,
|
||||
|
||||
# model settings
|
||||
"num-layers": 32,
|
||||
"hidden-size": 2560,
|
||||
"num-attention-heads": 32,
|
||||
"seq-length": 2048,
|
||||
"max-position-embeddings": 2048,
|
||||
|
||||
# architecture design
|
||||
"attention_head_type": "multihead",
|
||||
"norm": "layernorm",
|
||||
"pos-emb": "rotary",
|
||||
"rotary_pct": 0.25,
|
||||
"rotary_interleaved": false, # GPT-NeoX style
|
||||
"mlp_multiple_of": 256,
|
||||
"mlp_type": "gated",
|
||||
"activation": "silu",
|
||||
"no-weight-tying": true,
|
||||
"gpt_j_residual": true,
|
||||
"gpt_j_tied": true,
|
||||
"output_layer_parallelism": "column",
|
||||
|
||||
# biases
|
||||
"use_bias_in_norms": true,
|
||||
"use_bias_in_attn_linear": false,
|
||||
"use_bias_in_mlp": false,
|
||||
|
||||
# fused ops
|
||||
"bias-gelu-fusion": false,
|
||||
"scaled-upper-triang-masked-softmax-fusion": true,
|
||||
"attention-config": [[["flash"], 32]],
|
||||
|
||||
# optimizer settings
|
||||
"optimizer": {
|
||||
"type": "Adam",
|
||||
"params": {
|
||||
"lr": 3.2e-4,
|
||||
"betas": [0.9, 0.95],
|
||||
"eps": 1.0e-6
|
||||
},
|
||||
},
|
||||
"min_lr": 3.2e-5,
|
||||
"train-iters": 245_000,
|
||||
"lr-decay-iters": 245_000,
|
||||
"lr-decay-style": "cosine",
|
||||
"warmup": 0.01,
|
||||
|
||||
# for all zero_optimization options, see https://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training
|
||||
"zero_optimization": {
|
||||
"stage": 1,
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 1260000000,
|
||||
"overlap_comm": true,
|
||||
"reduce_scatter": true,
|
||||
"reduce_bucket_size": 1260000000,
|
||||
"contiguous_gradients": true,
|
||||
"cpu_offload": false,
|
||||
},
|
||||
|
||||
# batch / data settings
|
||||
"train_micro_batch_size_per_gpu": 16,
|
||||
"gradient_accumulation_steps": 1,
|
||||
"data-impl": "mmap",
|
||||
"eval-interval": 10_000,
|
||||
"eval-iters": 10,
|
||||
"eval_batch_size": 4,
|
||||
"eval_tasks": ["lambada_openai", "piqa"],
|
||||
|
||||
# activation checkpointing
|
||||
"checkpoint-activations": true,
|
||||
"checkpoint-num-layers": 1,
|
||||
"partition-activations": true,
|
||||
"synchronize-each-layer": true,
|
||||
|
||||
# regularization
|
||||
"gradient_clipping": 1.0,
|
||||
"weight-decay": 0.1,
|
||||
"hidden-dropout": 0,
|
||||
"attention-dropout": 0,
|
||||
|
||||
# precision settings
|
||||
"fp16": {
|
||||
"fp16": true,
|
||||
"enabled": true,
|
||||
"loss_scale": 0,
|
||||
# NOTE: Mid-training divergence required a loss scale of 1e-10
|
||||
# "loss_scale_window": 1000,
|
||||
# "initial_scale_power": 12,
|
||||
# "hysteresis": 2,
|
||||
# "min_loss_scale": 1
|
||||
"loss_scale_window": 1000,
|
||||
"initial_scale_power": 12,
|
||||
"hysteresis": 2,
|
||||
"min_loss_scale": 1e-10
|
||||
},
|
||||
"full_precision_lm_cross_entropy": true,
|
||||
|
||||
# misc. training settings
|
||||
"num-workers": 1,
|
||||
"distributed-backend": "nccl",
|
||||
|
||||
# checkpoint settings
|
||||
"checkpoint-factor": 2_000,
|
||||
"save": "",
|
||||
"load": "",
|
||||
"s3_path": "",
|
||||
|
||||
# data path settings
|
||||
"train-data-paths": [""],
|
||||
"train-data-weights": [1.0],
|
||||
"valid-data-paths": [""],
|
||||
"valid-data-weights": [1.0],
|
||||
"test-data-paths": [""],
|
||||
"test-data-weights": [1.0],
|
||||
|
||||
# tokenizer settings
|
||||
"tokenizer-type": "HFTokenizer",
|
||||
"vocab-file": "neox-tokenizer-vocab.json",
|
||||
|
||||
# log settings
|
||||
"log-interval": 10,
|
||||
"steps_per_print": 10,
|
||||
"wall_clock_breakdown": true,
|
||||
|
||||
"use_wandb": true,
|
||||
"wandb_host": "",
|
||||
"wandb_team": "",
|
||||
"wandb_project": "",
|
||||
"wandb_group": "3B",
|
||||
"wandb_name": "stablelm-base-alpha-3b-v2",
|
||||
# "wandb_id": "",
|
||||
# "wandb_resume": "must",
|
||||
|
||||
# multi-node launcher
|
||||
"launcher": "slurm",
|
||||
"deepspeed_slurm": true,
|
||||
}
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
{
|
||||
# parallelism settings
|
||||
"pipe-parallel-size": 1,
|
||||
"model-parallel-size": 2,
|
||||
|
||||
# model settings
|
||||
"num-layers": 32,
|
||||
"hidden-size": 4096,
|
||||
"num-attention-heads": 32,
|
||||
"seq-length": 4096,
|
||||
"max-position-embeddings": 4096,
|
||||
|
||||
# architecture design
|
||||
"attention_head_type": "multihead",
|
||||
"norm": "layernorm",
|
||||
"pos-emb": "rotary",
|
||||
"rotary_pct": 0.25,
|
||||
"rotary_interleaved": false, # GPT-NeoX style
|
||||
# NOTE: Linear Position Scaling degrades sample quality after 10B tokens - do not use yet.
|
||||
# "rotary_scaling_factor": 2, # 2048 -> 4096
|
||||
"mlp_multiple_of": 256,
|
||||
"mlp_type": "gated",
|
||||
"activation": "silu",
|
||||
"no-weight-tying": true,
|
||||
"gpt_j_residual": true,
|
||||
"gpt_j_tied": true,
|
||||
"output_layer_parallelism": "column",
|
||||
|
||||
# biases
|
||||
"use_bias_in_norms": true,
|
||||
"use_bias_in_attn_linear": false,
|
||||
"use_bias_in_mlp": false,
|
||||
|
||||
# fused ops
|
||||
"bias-gelu-fusion": false,
|
||||
"scaled-upper-triang-masked-softmax-fusion": true,
|
||||
"attention-config": [[["flash"], 32]],
|
||||
|
||||
# optimizer settings
|
||||
"optimizer": {
|
||||
"type": "Adam",
|
||||
"params": {
|
||||
"lr": 2.2e-5,
|
||||
"betas": [0.9, 0.95],
|
||||
"eps": 1.0e-6
|
||||
},
|
||||
},
|
||||
"min_lr": 2.2e-6,
|
||||
"train-iters": 45_000,
|
||||
"lr-decay-iters": 45_000,
|
||||
"lr-decay-style": "cosine",
|
||||
"warmup": 0.00,
|
||||
|
||||
# for all zero_optimization options, see https://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training
|
||||
"zero_optimization": {
|
||||
"stage": 1,
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 1260000000,
|
||||
"overlap_comm": true,
|
||||
"reduce_scatter": true,
|
||||
"reduce_bucket_size": 1260000000,
|
||||
"contiguous_gradients": true,
|
||||
"cpu_offload": false,
|
||||
},
|
||||
|
||||
# batch / data settings
|
||||
"train_micro_batch_size_per_gpu": 3,
|
||||
"gradient_accumulation_steps": 1,
|
||||
"data-impl": "mmap",
|
||||
"eval-interval": 4_000,
|
||||
"eval-iters": 10,
|
||||
"eval_batch_size": 2,
|
||||
"eval_tasks": ["lambada_openai", "piqa"],
|
||||
|
||||
# activation checkpointing
|
||||
"checkpoint-activations": true,
|
||||
"checkpoint-num-layers": 1,
|
||||
"partition-activations": true,
|
||||
"synchronize-each-layer": true,
|
||||
|
||||
# regularization
|
||||
"gradient_clipping": 1.0,
|
||||
"weight-decay": 0.01,
|
||||
"hidden-dropout": 0,
|
||||
"attention-dropout": 0,
|
||||
|
||||
# precision settings
|
||||
"fp16": {
|
||||
"fp16": true,
|
||||
"enabled": true,
|
||||
"loss_scale": 0,
|
||||
"loss_scale_window": 1000,
|
||||
"initial_scale_power": 12,
|
||||
"hysteresis": 2,
|
||||
"min_loss_scale": 1e-12
|
||||
},
|
||||
"full_precision_lm_cross_entropy": true,
|
||||
|
||||
# misc. training settings
|
||||
"num-workers": 1,
|
||||
"distributed-backend": "nccl",
|
||||
|
||||
# checkpoint settings
|
||||
"checkpoint-factor": 2_000,
|
||||
"save": "",
|
||||
"load": "",
|
||||
"s3_path": "",
|
||||
"iteration": 245_000,
|
||||
"finetune": true,
|
||||
"no_checkpoint_arg_validation": true,
|
||||
"override_lr_scheduler": true,
|
||||
|
||||
# data path settings
|
||||
"train-data-paths": [""],
|
||||
"train-data-weights": [1.0],
|
||||
"valid-data-paths": [""],
|
||||
"valid-data-weights": [1.0],
|
||||
"test-data-paths": [""],
|
||||
"test-data-weights": [1.0],
|
||||
|
||||
# tokenizer settings
|
||||
"tokenizer-type": "HFTokenizer",
|
||||
"vocab-file": "neox-tokenizer-vocab.json",
|
||||
|
||||
# log settings
|
||||
"log-interval": 10,
|
||||
"steps_per_print": 10,
|
||||
"wall_clock_breakdown": true,
|
||||
|
||||
"use_wandb": true,
|
||||
"wandb_host": "",
|
||||
"wandb_team": "",
|
||||
"wandb_project": "",
|
||||
"wandb_group": "7B",
|
||||
"wandb_name": "stablelm-base-alpha-7b-v2-4k-finetune",
|
||||
# "wandb_id": "",
|
||||
# "wandb_resume": "must",
|
||||
|
||||
# multi-node launcher
|
||||
"launcher": "slurm",
|
||||
"deepspeed_slurm": true,
|
||||
}
|
||||
Executable
+136
@@ -0,0 +1,136 @@
|
||||
{
|
||||
# parallelism settings
|
||||
"pipe-parallel-size": 1,
|
||||
"model-parallel-size": 2,
|
||||
|
||||
# model settings
|
||||
"num-layers": 32,
|
||||
"hidden-size": 4096,
|
||||
"num-attention-heads": 32,
|
||||
"seq-length": 2048,
|
||||
"max-position-embeddings": 2048,
|
||||
|
||||
# architecture design
|
||||
"attention_head_type": "multihead",
|
||||
"norm": "layernorm",
|
||||
"pos-emb": "rotary",
|
||||
"rotary_pct": 0.25,
|
||||
"rotary_interleaved": false, # GPT-NeoX style
|
||||
"mlp_multiple_of": 256,
|
||||
"mlp_type": "gated",
|
||||
"activation": "silu",
|
||||
"no-weight-tying": true,
|
||||
"gpt_j_residual": true,
|
||||
"gpt_j_tied": true,
|
||||
"output_layer_parallelism": "column",
|
||||
|
||||
# biases
|
||||
"use_bias_in_norms": true,
|
||||
"use_bias_in_attn_linear": false,
|
||||
"use_bias_in_mlp": false,
|
||||
|
||||
# fused ops
|
||||
"bias-gelu-fusion": false,
|
||||
"scaled-upper-triang-masked-softmax-fusion": true,
|
||||
"attention-config": [[["flash"], 32]],
|
||||
|
||||
# optimizer settings
|
||||
"optimizer": {
|
||||
"type": "Adam",
|
||||
"params": {
|
||||
"lr": 3.0e-4,
|
||||
"betas": [0.9, 0.95],
|
||||
"eps": 1.0e-6
|
||||
},
|
||||
},
|
||||
"min_lr": 3.0e-5,
|
||||
"train-iters": 245_000,
|
||||
"lr-decay-iters": 245_000,
|
||||
"lr-decay-style": "cosine",
|
||||
"warmup": 0.01,
|
||||
|
||||
# for all zero_optimization options, see https://www.deepspeed.ai/docs/config-json/#zero-optimizations-for-fp16-training
|
||||
"zero_optimization": {
|
||||
"stage": 1,
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 1260000000,
|
||||
"overlap_comm": true,
|
||||
"reduce_scatter": true,
|
||||
"reduce_bucket_size": 1260000000,
|
||||
"contiguous_gradients": true,
|
||||
"cpu_offload": false,
|
||||
},
|
||||
|
||||
# batch / data settings
|
||||
"train_micro_batch_size_per_gpu": 12,
|
||||
"gradient_accumulation_steps": 1,
|
||||
"data-impl": "mmap",
|
||||
"eval-interval": 10_000,
|
||||
"eval-iters": 10,
|
||||
"eval_batch_size": 2,
|
||||
"eval_tasks": ["lambada_openai", "piqa"],
|
||||
|
||||
# activation checkpointing
|
||||
"checkpoint-activations": true,
|
||||
"checkpoint-num-layers": 1,
|
||||
"partition-activations": true,
|
||||
"synchronize-each-layer": true,
|
||||
|
||||
# regularization
|
||||
"gradient_clipping": 1.0,
|
||||
"weight-decay": 0.1,
|
||||
"hidden-dropout": 0,
|
||||
"attention-dropout": 0,
|
||||
|
||||
# precision settings
|
||||
"fp16": {
|
||||
"fp16": true,
|
||||
"enabled": true,
|
||||
"loss_scale": 0,
|
||||
"loss_scale_window": 1000,
|
||||
"initial_scale_power": 12,
|
||||
"hysteresis": 2,
|
||||
"min_loss_scale": 1e-12
|
||||
},
|
||||
"full_precision_lm_cross_entropy": true,
|
||||
|
||||
# misc. training settings
|
||||
"num-workers": 1,
|
||||
"distributed-backend": "nccl",
|
||||
|
||||
# checkpoint settings
|
||||
"checkpoint-factor": 2_000,
|
||||
"save": "",
|
||||
"load": "",
|
||||
"s3_path": "",
|
||||
|
||||
# data path settings
|
||||
"train-data-paths": [""],
|
||||
"train-data-weights": [1.0],
|
||||
"valid-data-paths": [""],
|
||||
"valid-data-weights": [1.0],
|
||||
"test-data-paths": [""],
|
||||
"test-data-weights": [1.0],
|
||||
|
||||
# tokenizer settings
|
||||
"tokenizer-type": "HFTokenizer",
|
||||
"vocab-file": "neox-tokenizer-vocab.json",
|
||||
|
||||
# log settings
|
||||
"log-interval": 10,
|
||||
"steps_per_print": 10,
|
||||
"wall_clock_breakdown": true,
|
||||
|
||||
"use_wandb": true,
|
||||
"wandb_host": "",
|
||||
"wandb_team": "",
|
||||
"wandb_project": "",
|
||||
"wandb_group": "7B",
|
||||
"wandb_name": "stablelm-base-alpha-7b-v2",
|
||||
# "wandb_id": "",
|
||||
# "wandb_resume": "must",
|
||||
|
||||
# multi-node launcher
|
||||
"launcher": "slurm",
|
||||
"deepspeed_slurm": true,
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.30119453924914674,
|
||||
"acc_stderr": 0.013406741767847626,
|
||||
"acc_norm": 0.3293515358361775,
|
||||
"acc_norm_stderr": 0.013734057652635474
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.6346801346801347,
|
||||
"acc_stderr": 0.009880576614806924,
|
||||
"acc_norm": 0.5909090909090909,
|
||||
"acc_norm_stderr": 0.010088775152615782
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.6412844036697247,
|
||||
"acc_stderr": 0.008388668034059405
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.45429197371041624,
|
||||
"acc_stderr": 0.004968888130290072,
|
||||
"acc_norm": 0.5944035052778331,
|
||||
"acc_norm_stderr": 0.004900036261309038
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 5.00138268807375,
|
||||
"ppl_stderr": 0.11803810628354432,
|
||||
"acc": 0.6514651659227635,
|
||||
"acc_stderr": 0.0066386652033128745
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.238,
|
||||
"acc_stderr": 0.019064072958198446,
|
||||
"acc_norm": 0.348,
|
||||
"acc_norm_stderr": 0.02132372863280751
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7410228509249184,
|
||||
"acc_stderr": 0.0102209660314056,
|
||||
"acc_norm": 0.7404787812840044,
|
||||
"acc_norm_stderr": 0.010227939888173923
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.882,
|
||||
"acc_stderr": 0.010206869264381791,
|
||||
"acc_norm": 0.832,
|
||||
"acc_norm_stderr": 0.011828605831454262
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4094165813715456,
|
||||
"acc_stderr": 0.011126849576589028,
|
||||
"acc_norm": 0.44319344933469806,
|
||||
"acc_norm_stderr": 0.011240812731564954
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.2141982864137087,
|
||||
"mc1_stderr": 0.014362148155690466,
|
||||
"mc2": 0.3555711185495532,
|
||||
"mc2_stderr": 0.013587679864140447
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.5824782951854776,
|
||||
"acc_stderr": 0.01385997826444025
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=EleutherAI/pythia-2.8b-deduped,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto","num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:4",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.34897610921501704,
|
||||
"acc_stderr": 0.013928933461382496,
|
||||
"acc_norm": 0.37627986348122866,
|
||||
"acc_norm_stderr": 0.014157022555407168
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.7045454545454546,
|
||||
"acc_stderr": 0.009361987126556457,
|
||||
"acc_norm": 0.6708754208754208,
|
||||
"acc_norm_stderr": 0.00964204805806098
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.6963302752293578,
|
||||
"acc_stderr": 0.008042682539896304
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5184226249751046,
|
||||
"acc_stderr": 0.004986393266269161,
|
||||
"acc_norm": 0.6977693686516631,
|
||||
"acc_norm_stderr": 0.004582861219020893
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 4.720441000893734,
|
||||
"ppl_stderr": 0.1100556659950519,
|
||||
"acc": 0.6623326217737241,
|
||||
"acc_stderr": 0.00658862361668043
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.276,
|
||||
"acc_stderr": 0.02001121929807353,
|
||||
"acc_norm": 0.408,
|
||||
"acc_norm_stderr": 0.02200091089387719
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7584330794341676,
|
||||
"acc_stderr": 0.009986718001804472,
|
||||
"acc_norm": 0.7720348204570185,
|
||||
"acc_norm_stderr": 0.009788093832324906
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.929,
|
||||
"acc_stderr": 0.008125578442487916,
|
||||
"acc_norm": 0.895,
|
||||
"acc_norm_stderr": 0.009698921026024966
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4278403275332651,
|
||||
"acc_stderr": 0.01119562541819821,
|
||||
"acc_norm": 0.4611054247697032,
|
||||
"acc_norm_stderr": 0.011279787032703659
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.2252141982864137,
|
||||
"mc1_stderr": 0.014623240768023498,
|
||||
"mc2": 0.3599562107238256,
|
||||
"mc2_stderr": 0.013576568348894856
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6495659037095501,
|
||||
"acc_stderr": 0.013409047676670189
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=cerebras/btlm-3b-8k-base,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto","num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:4",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.4300341296928328,
|
||||
"acc_stderr": 0.014467631559137993,
|
||||
"acc_norm": 0.4616040955631399,
|
||||
"acc_norm_stderr": 0.014568245550296363
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.7626262626262627,
|
||||
"acc_stderr": 0.008730525906362434,
|
||||
"acc_norm": 0.7453703703703703,
|
||||
"acc_norm_stderr": 0.008939407288589414
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.7773700305810397,
|
||||
"acc_stderr": 0.007276093141006333
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5720971917944633,
|
||||
"acc_stderr": 0.00493763511283029,
|
||||
"acc_norm": 0.7594104760007967,
|
||||
"acc_norm_stderr": 0.004265678940698863
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 3.3970918836338027,
|
||||
"ppl_stderr": 0.06684659102563836,
|
||||
"acc": 0.7347176402095866,
|
||||
"acc_stderr": 0.0061507275830540355
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.314,
|
||||
"acc_stderr": 0.020776701920308997,
|
||||
"acc_norm": 0.436,
|
||||
"acc_norm_stderr": 0.0221989546414768
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7774755168661589,
|
||||
"acc_stderr": 0.009704600975718245,
|
||||
"acc_norm": 0.7878128400435256,
|
||||
"acc_norm_stderr": 0.009539299828174048
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.936,
|
||||
"acc_stderr": 0.007743640226919308,
|
||||
"acc_norm": 0.908,
|
||||
"acc_norm_stderr": 0.009144376393151106
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.43500511770726713,
|
||||
"acc_stderr": 0.011218074465506494,
|
||||
"acc_norm": 0.47389969293756395,
|
||||
"acc_norm_stderr": 0.011298645160980832
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.2521419828641371,
|
||||
"mc1_stderr": 0.01520152224629997,
|
||||
"mc2": 0.38967559882659686,
|
||||
"mc2_stderr": 0.01357922221561985
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6961325966850829,
|
||||
"acc_stderr": 0.012926209475483574
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=False,pretrained=meta-llama/Llama-2-7b-hf,trust_remote_code=True,low_cpu_mem_usage=True,dtype=bfloat16",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "2",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:0",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
Vendored
+91
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.4052901023890785,
|
||||
"acc_stderr": 0.01434686906022933,
|
||||
"acc_norm": 0.4189419795221843,
|
||||
"acc_norm_stderr": 0.014418106953639011
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.7491582491582491,
|
||||
"acc_stderr": 0.008895183010487386,
|
||||
"acc_norm": 0.7003367003367004,
|
||||
"acc_norm_stderr": 0.009400228586205971
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.7394495412844037,
|
||||
"acc_stderr": 0.007677021072511165
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.571400119498108,
|
||||
"acc_stderr": 0.004938643787869543,
|
||||
"acc_norm": 0.7617008564031069,
|
||||
"acc_norm_stderr": 0.00425172316377217
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 3.8689102314884214,
|
||||
"ppl_stderr": 0.0808940319922043,
|
||||
"acc": 0.6863962740151368,
|
||||
"acc_stderr": 0.006463833164285203
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.314,
|
||||
"acc_stderr": 0.020776701920308997,
|
||||
"acc_norm": 0.428,
|
||||
"acc_norm_stderr": 0.022149790663861923
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7889009793253536,
|
||||
"acc_stderr": 0.00952137737873414,
|
||||
"acc_norm": 0.8063112078346029,
|
||||
"acc_norm_stderr": 0.009220384152336641
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.937,
|
||||
"acc_stderr": 0.007687007876286428,
|
||||
"acc_norm": 0.888,
|
||||
"acc_norm_stderr": 0.009977753031397236
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.45138178096212894,
|
||||
"acc_stderr": 0.01126045668162444,
|
||||
"acc_norm": 0.48311156601842375,
|
||||
"acc_norm_stderr": 0.011307614732827416
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.20930232558139536,
|
||||
"mc1_stderr": 0.014241219434785823,
|
||||
"mc2": 0.3348523259251629,
|
||||
"mc2_stderr": 0.01313621094524683
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6803472770323599,
|
||||
"acc_stderr": 0.01310652851766514
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=mosaicml/mpt-7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:2",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.3387372013651877,
|
||||
"acc_stderr": 0.013830568927974332,
|
||||
"acc_norm": 0.3609215017064846,
|
||||
"acc_norm_stderr": 0.014034761386175458
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.6759259259259259,
|
||||
"acc_stderr": 0.00960372885009539,
|
||||
"acc_norm": 0.63510101010101,
|
||||
"acc_norm_stderr": 0.009878157021155649
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.6568807339449542,
|
||||
"acc_stderr": 0.008303445777655941
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5223063134833699,
|
||||
"acc_stderr": 0.0049848133910162145,
|
||||
"acc_norm": 0.6998605855407289,
|
||||
"acc_norm_stderr": 0.004573817163007456
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 4.565625743504039,
|
||||
"ppl_stderr": 0.1034965673734545,
|
||||
"acc": 0.6673782262759558,
|
||||
"acc_stderr": 0.006564073374961233
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.26,
|
||||
"acc_stderr": 0.019635965529725512,
|
||||
"acc_norm": 0.376,
|
||||
"acc_norm_stderr": 0.021683827539286122
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7665941240478781,
|
||||
"acc_stderr": 0.009869247889521001,
|
||||
"acc_norm": 0.778563656147987,
|
||||
"acc_norm_stderr": 0.009687616456840284
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.924,
|
||||
"acc_stderr": 0.008384169266796386,
|
||||
"acc_norm": 0.878,
|
||||
"acc_norm_stderr": 0.010354864712936698
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4119754350051177,
|
||||
"acc_stderr": 0.011137360400975268,
|
||||
"acc_norm": 0.4524053224155578,
|
||||
"acc_norm_stderr": 0.011262695440459566
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.21297429620563035,
|
||||
"mc1_stderr": 0.014332203787059685,
|
||||
"mc2": 0.3458747299959986,
|
||||
"mc2_stderr": 0.013215129281312441
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6290449881610103,
|
||||
"acc_stderr": 0.01357639990223157
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=False,pretrained=openlm-research/open_llama_3b_v2,trust_remote_code=True,low_cpu_mem_usage=True,dtype=bfloat16",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:4",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.38822525597269625,
|
||||
"acc_stderr": 0.014241614207414034,
|
||||
"acc_norm": 0.42406143344709896,
|
||||
"acc_norm_stderr": 0.014441889627464392
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.7192760942760943,
|
||||
"acc_stderr": 0.009220526174711363,
|
||||
"acc_norm": 0.6965488215488216,
|
||||
"acc_norm_stderr": 0.009433837434252279
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.7140672782874617,
|
||||
"acc_stderr": 0.00790303735916362
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5569607647878908,
|
||||
"acc_stderr": 0.004957296691391575,
|
||||
"acc_norm": 0.7464648476399124,
|
||||
"acc_norm_stderr": 0.004341454841892331
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 3.8265934555236463,
|
||||
"ppl_stderr": 0.07878439404908999,
|
||||
"acc": 0.7104599262565496,
|
||||
"acc_stderr": 0.006318823234213216
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.302,
|
||||
"acc_stderr": 0.020553269174209184,
|
||||
"acc_norm": 0.402,
|
||||
"acc_norm_stderr": 0.021948929609938606
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7916213275299239,
|
||||
"acc_stderr": 0.009476125383049447,
|
||||
"acc_norm": 0.8030467899891186,
|
||||
"acc_norm_stderr": 0.009278918898006383
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.938,
|
||||
"acc_stderr": 0.007629823996280304,
|
||||
"acc_norm": 0.901,
|
||||
"acc_norm_stderr": 0.009449248027662747
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4196519959058342,
|
||||
"acc_stderr": 0.011167032303390547,
|
||||
"acc_norm": 0.4600818833162743,
|
||||
"acc_norm_stderr": 0.011277955967920398
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.22643818849449204,
|
||||
"mc1_stderr": 0.014651337324602576,
|
||||
"mc2": 0.3456877328963021,
|
||||
"mc2_stderr": 0.013482248222806824
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6582478295185478,
|
||||
"acc_stderr": 0.013330103018622861
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=False,pretrained=openlm-research/open_llama_7b_v2,trust_remote_code=True,low_cpu_mem_usage=True,dtype=bfloat16",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "2",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:0",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.3771331058020478,
|
||||
"acc_stderr": 0.014163366896192593,
|
||||
"acc_norm": 0.39419795221843,
|
||||
"acc_norm_stderr": 0.014280522667467323
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.7234848484848485,
|
||||
"acc_stderr": 0.00917788010146828,
|
||||
"acc_norm": 0.6919191919191919,
|
||||
"acc_norm_stderr": 0.009473887075826333
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.7076452599388379,
|
||||
"acc_stderr": 0.00795527890990574
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5256920932085242,
|
||||
"acc_stderr": 0.004983189711208505,
|
||||
"acc_norm": 0.7033459470225055,
|
||||
"acc_norm_stderr": 0.004558491550673688
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 3.9176883876529036,
|
||||
"ppl_stderr": 0.08360630042095321,
|
||||
"acc": 0.713370851930914,
|
||||
"acc_stderr": 0.006299845944000654
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.29,
|
||||
"acc_stderr": 0.020313179231745186,
|
||||
"acc_norm": 0.406,
|
||||
"acc_norm_stderr": 0.021983962090086333
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7714907508161044,
|
||||
"acc_stderr": 0.009796313511829524,
|
||||
"acc_norm": 0.7736670293797606,
|
||||
"acc_norm_stderr": 0.009763294246879415
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.927,
|
||||
"acc_stderr": 0.008230354715244055,
|
||||
"acc_norm": 0.897,
|
||||
"acc_norm_stderr": 0.009616833339695796
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4257932446264074,
|
||||
"acc_stderr": 0.011188771652377858,
|
||||
"acc_norm": 0.44882292732855683,
|
||||
"acc_norm_stderr": 0.011254649314820132
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.23011015911872704,
|
||||
"mc1_stderr": 0.014734557959807769,
|
||||
"mc2": 0.3301333660822527,
|
||||
"mc2_stderr": 0.012995959559501016
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6432517758484609,
|
||||
"acc_stderr": 0.013463393958028733
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=togethercomputer/RedPajama-INCITE-7B-Base,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:2",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"results": {
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.23745410036719705,
|
||||
"mc1_stderr": 0.014896277441041836,
|
||||
"mc2": 0.3995908363542637,
|
||||
"mc2_stderr": 0.014371652685680641
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"truthfulqa_mc": 1
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=stabilityai/stablelm-base-alpha-7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:3",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.36689419795221845,
|
||||
"acc_norm": 0.3967576791808874
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-3b-v2",
|
||||
"num_fewshot": 25,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"results": {
|
||||
"hellaswag": {
|
||||
"acc": 0.5239992033459471,
|
||||
"acc_norm": 0.7066321449910377
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"hellaswag": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-3b-v2",
|
||||
"num_fewshot": 10,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,301 @@
|
||||
{
|
||||
"results": {
|
||||
"hendrycksTest-professional_psychology": {
|
||||
"acc": 0.2826797385620915,
|
||||
"acc_norm": 0.27124183006535946
|
||||
},
|
||||
"hendrycksTest-abstract_algebra": {
|
||||
"acc": 0.26,
|
||||
"acc_norm": 0.29
|
||||
},
|
||||
"hendrycksTest-prehistory": {
|
||||
"acc": 0.3271604938271605,
|
||||
"acc_norm": 0.26851851851851855
|
||||
},
|
||||
"hendrycksTest-international_law": {
|
||||
"acc": 0.23140495867768596,
|
||||
"acc_norm": 0.36363636363636365
|
||||
},
|
||||
"hendrycksTest-logical_fallacies": {
|
||||
"acc": 0.20245398773006135,
|
||||
"acc_norm": 0.22085889570552147
|
||||
},
|
||||
"hendrycksTest-professional_medicine": {
|
||||
"acc": 0.30514705882352944,
|
||||
"acc_norm": 0.33455882352941174
|
||||
},
|
||||
"hendrycksTest-high_school_european_history": {
|
||||
"acc": 0.3151515151515151,
|
||||
"acc_norm": 0.296969696969697
|
||||
},
|
||||
"hendrycksTest-high_school_physics": {
|
||||
"acc": 0.2251655629139073,
|
||||
"acc_norm": 0.2582781456953642
|
||||
},
|
||||
"hendrycksTest-management": {
|
||||
"acc": 0.30097087378640774,
|
||||
"acc_norm": 0.30097087378640774
|
||||
},
|
||||
"hendrycksTest-college_mathematics": {
|
||||
"acc": 0.17,
|
||||
"acc_norm": 0.24
|
||||
},
|
||||
"hendrycksTest-college_computer_science": {
|
||||
"acc": 0.29,
|
||||
"acc_norm": 0.24
|
||||
},
|
||||
"hendrycksTest-human_sexuality": {
|
||||
"acc": 0.40458015267175573,
|
||||
"acc_norm": 0.3893129770992366
|
||||
},
|
||||
"hendrycksTest-college_biology": {
|
||||
"acc": 0.2708333333333333,
|
||||
"acc_norm": 0.2569444444444444
|
||||
},
|
||||
"hendrycksTest-high_school_computer_science": {
|
||||
"acc": 0.34,
|
||||
"acc_norm": 0.3
|
||||
},
|
||||
"hendrycksTest-high_school_psychology": {
|
||||
"acc": 0.3467889908256881,
|
||||
"acc_norm": 0.326605504587156
|
||||
},
|
||||
"hendrycksTest-high_school_chemistry": {
|
||||
"acc": 0.2413793103448276,
|
||||
"acc_norm": 0.28078817733990147
|
||||
},
|
||||
"hendrycksTest-astronomy": {
|
||||
"acc": 0.3355263157894737,
|
||||
"acc_norm": 0.3355263157894737
|
||||
},
|
||||
"hendrycksTest-medical_genetics": {
|
||||
"acc": 0.39,
|
||||
"acc_norm": 0.4
|
||||
},
|
||||
"hendrycksTest-nutrition": {
|
||||
"acc": 0.35947712418300654,
|
||||
"acc_norm": 0.3758169934640523
|
||||
},
|
||||
"hendrycksTest-moral_disputes": {
|
||||
"acc": 0.3554913294797688,
|
||||
"acc_norm": 0.33815028901734107
|
||||
},
|
||||
"hendrycksTest-computer_security": {
|
||||
"acc": 0.38,
|
||||
"acc_norm": 0.39
|
||||
},
|
||||
"hendrycksTest-anatomy": {
|
||||
"acc": 0.32592592592592595,
|
||||
"acc_norm": 0.2814814814814815
|
||||
},
|
||||
"hendrycksTest-formal_logic": {
|
||||
"acc": 0.30952380952380953,
|
||||
"acc_norm": 0.3253968253968254
|
||||
},
|
||||
"hendrycksTest-high_school_us_history": {
|
||||
"acc": 0.3088235294117647,
|
||||
"acc_norm": 0.29901960784313725
|
||||
},
|
||||
"hendrycksTest-security_studies": {
|
||||
"acc": 0.3224489795918367,
|
||||
"acc_norm": 0.23265306122448978
|
||||
},
|
||||
"hendrycksTest-high_school_mathematics": {
|
||||
"acc": 0.23703703703703705,
|
||||
"acc_norm": 0.2814814814814815
|
||||
},
|
||||
"hendrycksTest-high_school_macroeconomics": {
|
||||
"acc": 0.2794871794871795,
|
||||
"acc_norm": 0.2794871794871795
|
||||
},
|
||||
"hendrycksTest-clinical_knowledge": {
|
||||
"acc": 0.26037735849056604,
|
||||
"acc_norm": 0.32075471698113206
|
||||
},
|
||||
"hendrycksTest-us_foreign_policy": {
|
||||
"acc": 0.39,
|
||||
"acc_norm": 0.41
|
||||
},
|
||||
"hendrycksTest-virology": {
|
||||
"acc": 0.35542168674698793,
|
||||
"acc_norm": 0.3493975903614458
|
||||
},
|
||||
"hendrycksTest-public_relations": {
|
||||
"acc": 0.34545454545454546,
|
||||
"acc_norm": 0.3
|
||||
},
|
||||
"hendrycksTest-world_religions": {
|
||||
"acc": 0.4093567251461988,
|
||||
"acc_norm": 0.4619883040935672
|
||||
},
|
||||
"hendrycksTest-college_physics": {
|
||||
"acc": 0.3137254901960784,
|
||||
"acc_norm": 0.3235294117647059
|
||||
},
|
||||
"hendrycksTest-high_school_biology": {
|
||||
"acc": 0.267741935483871,
|
||||
"acc_norm": 0.2870967741935484
|
||||
},
|
||||
"hendrycksTest-business_ethics": {
|
||||
"acc": 0.33,
|
||||
"acc_norm": 0.28
|
||||
},
|
||||
"hendrycksTest-high_school_government_and_politics": {
|
||||
"acc": 0.30569948186528495,
|
||||
"acc_norm": 0.2849740932642487
|
||||
},
|
||||
"hendrycksTest-high_school_world_history": {
|
||||
"acc": 0.2742616033755274,
|
||||
"acc_norm": 0.27848101265822783
|
||||
},
|
||||
"hendrycksTest-jurisprudence": {
|
||||
"acc": 0.23148148148148148,
|
||||
"acc_norm": 0.24074074074074073
|
||||
},
|
||||
"hendrycksTest-miscellaneous": {
|
||||
"acc": 0.4227330779054917,
|
||||
"acc_norm": 0.41762452107279696
|
||||
},
|
||||
"hendrycksTest-marketing": {
|
||||
"acc": 0.3974358974358974,
|
||||
"acc_norm": 0.39316239316239315
|
||||
},
|
||||
"hendrycksTest-high_school_microeconomics": {
|
||||
"acc": 0.29411764705882354,
|
||||
"acc_norm": 0.3445378151260504
|
||||
},
|
||||
"hendrycksTest-econometrics": {
|
||||
"acc": 0.2543859649122807,
|
||||
"acc_norm": 0.24561403508771928
|
||||
},
|
||||
"hendrycksTest-conceptual_physics": {
|
||||
"acc": 0.3276595744680851,
|
||||
"acc_norm": 0.28936170212765955
|
||||
},
|
||||
"hendrycksTest-high_school_statistics": {
|
||||
"acc": 0.3287037037037037,
|
||||
"acc_norm": 0.33796296296296297
|
||||
},
|
||||
"hendrycksTest-sociology": {
|
||||
"acc": 0.3333333333333333,
|
||||
"acc_norm": 0.31343283582089554
|
||||
},
|
||||
"hendrycksTest-electrical_engineering": {
|
||||
"acc": 0.2689655172413793,
|
||||
"acc_norm": 0.25517241379310346
|
||||
},
|
||||
"hendrycksTest-elementary_mathematics": {
|
||||
"acc": 0.335978835978836,
|
||||
"acc_norm": 0.328042328042328
|
||||
},
|
||||
"hendrycksTest-high_school_geography": {
|
||||
"acc": 0.3181818181818182,
|
||||
"acc_norm": 0.3333333333333333
|
||||
},
|
||||
"hendrycksTest-philosophy": {
|
||||
"acc": 0.31511254019292606,
|
||||
"acc_norm": 0.3247588424437299
|
||||
},
|
||||
"hendrycksTest-moral_scenarios": {
|
||||
"acc": 0.26927374301675977,
|
||||
"acc_norm": 0.27150837988826815
|
||||
},
|
||||
"hendrycksTest-college_chemistry": {
|
||||
"acc": 0.34,
|
||||
"acc_norm": 0.36
|
||||
},
|
||||
"hendrycksTest-machine_learning": {
|
||||
"acc": 0.2767857142857143,
|
||||
"acc_norm": 0.2767857142857143
|
||||
},
|
||||
"hendrycksTest-professional_accounting": {
|
||||
"acc": 0.29432624113475175,
|
||||
"acc_norm": 0.3049645390070922
|
||||
},
|
||||
"hendrycksTest-professional_law": {
|
||||
"acc": 0.25488917861799215,
|
||||
"acc_norm": 0.258148631029987
|
||||
},
|
||||
"hendrycksTest-college_medicine": {
|
||||
"acc": 0.35260115606936415,
|
||||
"acc_norm": 0.3468208092485549
|
||||
},
|
||||
"hendrycksTest-global_facts": {
|
||||
"acc": 0.26,
|
||||
"acc_norm": 0.21
|
||||
},
|
||||
"hendrycksTest-human_aging": {
|
||||
"acc": 0.28699551569506726,
|
||||
"acc_norm": 0.2556053811659193
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"hendrycksTest-professional_psychology": 0,
|
||||
"hendrycksTest-abstract_algebra": 0,
|
||||
"hendrycksTest-prehistory": 0,
|
||||
"hendrycksTest-international_law": 0,
|
||||
"hendrycksTest-logical_fallacies": 0,
|
||||
"hendrycksTest-professional_medicine": 0,
|
||||
"hendrycksTest-high_school_european_history": 0,
|
||||
"hendrycksTest-high_school_physics": 0,
|
||||
"hendrycksTest-management": 0,
|
||||
"hendrycksTest-college_mathematics": 0,
|
||||
"hendrycksTest-college_computer_science": 0,
|
||||
"hendrycksTest-human_sexuality": 0,
|
||||
"hendrycksTest-college_biology": 0,
|
||||
"hendrycksTest-high_school_computer_science": 0,
|
||||
"hendrycksTest-high_school_psychology": 0,
|
||||
"hendrycksTest-high_school_chemistry": 0,
|
||||
"hendrycksTest-astronomy": 0,
|
||||
"hendrycksTest-medical_genetics": 0,
|
||||
"hendrycksTest-nutrition": 0,
|
||||
"hendrycksTest-moral_disputes": 0,
|
||||
"hendrycksTest-computer_security": 0,
|
||||
"hendrycksTest-anatomy": 0,
|
||||
"hendrycksTest-formal_logic": 0,
|
||||
"hendrycksTest-high_school_us_history": 0,
|
||||
"hendrycksTest-security_studies": 0,
|
||||
"hendrycksTest-high_school_mathematics": 0,
|
||||
"hendrycksTest-high_school_macroeconomics": 0,
|
||||
"hendrycksTest-clinical_knowledge": 0,
|
||||
"hendrycksTest-us_foreign_policy": 0,
|
||||
"hendrycksTest-virology": 0,
|
||||
"hendrycksTest-public_relations": 0,
|
||||
"hendrycksTest-world_religions": 0,
|
||||
"hendrycksTest-college_physics": 0,
|
||||
"hendrycksTest-high_school_biology": 0,
|
||||
"hendrycksTest-business_ethics": 0,
|
||||
"hendrycksTest-high_school_government_and_politics": 0,
|
||||
"hendrycksTest-high_school_world_history": 0,
|
||||
"hendrycksTest-jurisprudence": 0,
|
||||
"hendrycksTest-miscellaneous": 0,
|
||||
"hendrycksTest-marketing": 0,
|
||||
"hendrycksTest-high_school_microeconomics": 0,
|
||||
"hendrycksTest-econometrics": 0,
|
||||
"hendrycksTest-conceptual_physics": 0,
|
||||
"hendrycksTest-high_school_statistics": 0,
|
||||
"hendrycksTest-sociology": 0,
|
||||
"hendrycksTest-electrical_engineering": 0,
|
||||
"hendrycksTest-elementary_mathematics": 0,
|
||||
"hendrycksTest-high_school_geography": 0,
|
||||
"hendrycksTest-philosophy": 0,
|
||||
"hendrycksTest-moral_scenarios": 0,
|
||||
"hendrycksTest-college_chemistry": 0,
|
||||
"hendrycksTest-machine_learning": 0,
|
||||
"hendrycksTest-professional_accounting": 0,
|
||||
"hendrycksTest-professional_law": 0,
|
||||
"hendrycksTest-college_medicine": 0,
|
||||
"hendrycksTest-global_facts": 0,
|
||||
"hendrycksTest-human_aging": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-3b-v2",
|
||||
"num_fewshot": 5,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"results": {
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.22399020807833536,
|
||||
"mc2": 0.35873137294835583
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"truthfulqa_mc": 1
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-3b-v2",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.431740614334471,
|
||||
"acc_norm": 0.4726962457337884
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-7b-v2",
|
||||
"num_fewshot": 25,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"results": {
|
||||
"hellaswag": {
|
||||
"acc": 0.5721967735510854,
|
||||
"acc_norm": 0.7706632144991038
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"hellaswag": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-7b-v2",
|
||||
"num_fewshot": 10,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,301 @@
|
||||
{
|
||||
"results": {
|
||||
"hendrycksTest-high_school_european_history": {
|
||||
"acc": 0.48484848484848486,
|
||||
"acc_norm": 0.47878787878787876
|
||||
},
|
||||
"hendrycksTest-college_chemistry": {
|
||||
"acc": 0.29,
|
||||
"acc_norm": 0.28
|
||||
},
|
||||
"hendrycksTest-international_law": {
|
||||
"acc": 0.38016528925619836,
|
||||
"acc_norm": 0.4214876033057851
|
||||
},
|
||||
"hendrycksTest-high_school_macroeconomics": {
|
||||
"acc": 0.36153846153846153,
|
||||
"acc_norm": 0.35128205128205126
|
||||
},
|
||||
"hendrycksTest-miscellaneous": {
|
||||
"acc": 0.6513409961685823,
|
||||
"acc_norm": 0.644955300127714
|
||||
},
|
||||
"hendrycksTest-professional_law": {
|
||||
"acc": 0.31877444589308995,
|
||||
"acc_norm": 0.3116036505867014
|
||||
},
|
||||
"hendrycksTest-medical_genetics": {
|
||||
"acc": 0.51,
|
||||
"acc_norm": 0.55
|
||||
},
|
||||
"hendrycksTest-high_school_world_history": {
|
||||
"acc": 0.3881856540084388,
|
||||
"acc_norm": 0.4050632911392405
|
||||
},
|
||||
"hendrycksTest-professional_medicine": {
|
||||
"acc": 0.4742647058823529,
|
||||
"acc_norm": 0.44485294117647056
|
||||
},
|
||||
"hendrycksTest-moral_disputes": {
|
||||
"acc": 0.42196531791907516,
|
||||
"acc_norm": 0.430635838150289
|
||||
},
|
||||
"hendrycksTest-high_school_geography": {
|
||||
"acc": 0.5404040404040404,
|
||||
"acc_norm": 0.5454545454545454
|
||||
},
|
||||
"hendrycksTest-high_school_microeconomics": {
|
||||
"acc": 0.42857142857142855,
|
||||
"acc_norm": 0.42436974789915966
|
||||
},
|
||||
"hendrycksTest-machine_learning": {
|
||||
"acc": 0.32142857142857145,
|
||||
"acc_norm": 0.32142857142857145
|
||||
},
|
||||
"hendrycksTest-security_studies": {
|
||||
"acc": 0.3510204081632653,
|
||||
"acc_norm": 0.2612244897959184
|
||||
},
|
||||
"hendrycksTest-world_religions": {
|
||||
"acc": 0.6608187134502924,
|
||||
"acc_norm": 0.695906432748538
|
||||
},
|
||||
"hendrycksTest-conceptual_physics": {
|
||||
"acc": 0.40425531914893614,
|
||||
"acc_norm": 0.3659574468085106
|
||||
},
|
||||
"hendrycksTest-high_school_physics": {
|
||||
"acc": 0.2582781456953642,
|
||||
"acc_norm": 0.2980132450331126
|
||||
},
|
||||
"hendrycksTest-nutrition": {
|
||||
"acc": 0.43137254901960786,
|
||||
"acc_norm": 0.45098039215686275
|
||||
},
|
||||
"hendrycksTest-high_school_psychology": {
|
||||
"acc": 0.581651376146789,
|
||||
"acc_norm": 0.5614678899082569
|
||||
},
|
||||
"hendrycksTest-professional_accounting": {
|
||||
"acc": 0.2907801418439716,
|
||||
"acc_norm": 0.30141843971631205
|
||||
},
|
||||
"hendrycksTest-human_aging": {
|
||||
"acc": 0.4304932735426009,
|
||||
"acc_norm": 0.39461883408071746
|
||||
},
|
||||
"hendrycksTest-college_physics": {
|
||||
"acc": 0.35294117647058826,
|
||||
"acc_norm": 0.3627450980392157
|
||||
},
|
||||
"hendrycksTest-high_school_chemistry": {
|
||||
"acc": 0.28078817733990147,
|
||||
"acc_norm": 0.3054187192118227
|
||||
},
|
||||
"hendrycksTest-high_school_biology": {
|
||||
"acc": 0.4290322580645161,
|
||||
"acc_norm": 0.43548387096774194
|
||||
},
|
||||
"hendrycksTest-us_foreign_policy": {
|
||||
"acc": 0.57,
|
||||
"acc_norm": 0.53
|
||||
},
|
||||
"hendrycksTest-philosophy": {
|
||||
"acc": 0.4758842443729904,
|
||||
"acc_norm": 0.4662379421221865
|
||||
},
|
||||
"hendrycksTest-logical_fallacies": {
|
||||
"acc": 0.4110429447852761,
|
||||
"acc_norm": 0.4294478527607362
|
||||
},
|
||||
"hendrycksTest-anatomy": {
|
||||
"acc": 0.48148148148148145,
|
||||
"acc_norm": 0.4888888888888889
|
||||
},
|
||||
"hendrycksTest-jurisprudence": {
|
||||
"acc": 0.4351851851851852,
|
||||
"acc_norm": 0.5
|
||||
},
|
||||
"hendrycksTest-high_school_computer_science": {
|
||||
"acc": 0.41,
|
||||
"acc_norm": 0.46
|
||||
},
|
||||
"hendrycksTest-elementary_mathematics": {
|
||||
"acc": 0.32275132275132273,
|
||||
"acc_norm": 0.32275132275132273
|
||||
},
|
||||
"hendrycksTest-abstract_algebra": {
|
||||
"acc": 0.27,
|
||||
"acc_norm": 0.28
|
||||
},
|
||||
"hendrycksTest-prehistory": {
|
||||
"acc": 0.44753086419753085,
|
||||
"acc_norm": 0.44135802469135804
|
||||
},
|
||||
"hendrycksTest-moral_scenarios": {
|
||||
"acc": 0.2581005586592179,
|
||||
"acc_norm": 0.26033519553072626
|
||||
},
|
||||
"hendrycksTest-college_medicine": {
|
||||
"acc": 0.4161849710982659,
|
||||
"acc_norm": 0.41040462427745666
|
||||
},
|
||||
"hendrycksTest-econometrics": {
|
||||
"acc": 0.2982456140350877,
|
||||
"acc_norm": 0.2807017543859649
|
||||
},
|
||||
"hendrycksTest-human_sexuality": {
|
||||
"acc": 0.5038167938931297,
|
||||
"acc_norm": 0.4961832061068702
|
||||
},
|
||||
"hendrycksTest-management": {
|
||||
"acc": 0.5242718446601942,
|
||||
"acc_norm": 0.5339805825242718
|
||||
},
|
||||
"hendrycksTest-computer_security": {
|
||||
"acc": 0.57,
|
||||
"acc_norm": 0.54
|
||||
},
|
||||
"hendrycksTest-college_computer_science": {
|
||||
"acc": 0.38,
|
||||
"acc_norm": 0.34
|
||||
},
|
||||
"hendrycksTest-public_relations": {
|
||||
"acc": 0.5636363636363636,
|
||||
"acc_norm": 0.5454545454545454
|
||||
},
|
||||
"hendrycksTest-sociology": {
|
||||
"acc": 0.5522388059701493,
|
||||
"acc_norm": 0.5124378109452736
|
||||
},
|
||||
"hendrycksTest-global_facts": {
|
||||
"acc": 0.29,
|
||||
"acc_norm": 0.28
|
||||
},
|
||||
"hendrycksTest-astronomy": {
|
||||
"acc": 0.42105263157894735,
|
||||
"acc_norm": 0.42105263157894735
|
||||
},
|
||||
"hendrycksTest-high_school_statistics": {
|
||||
"acc": 0.27314814814814814,
|
||||
"acc_norm": 0.2916666666666667
|
||||
},
|
||||
"hendrycksTest-professional_psychology": {
|
||||
"acc": 0.40522875816993464,
|
||||
"acc_norm": 0.39052287581699346
|
||||
},
|
||||
"hendrycksTest-high_school_us_history": {
|
||||
"acc": 0.37745098039215685,
|
||||
"acc_norm": 0.39705882352941174
|
||||
},
|
||||
"hendrycksTest-business_ethics": {
|
||||
"acc": 0.42,
|
||||
"acc_norm": 0.44
|
||||
},
|
||||
"hendrycksTest-clinical_knowledge": {
|
||||
"acc": 0.42641509433962266,
|
||||
"acc_norm": 0.4226415094339623
|
||||
},
|
||||
"hendrycksTest-college_biology": {
|
||||
"acc": 0.4236111111111111,
|
||||
"acc_norm": 0.4166666666666667
|
||||
},
|
||||
"hendrycksTest-formal_logic": {
|
||||
"acc": 0.42063492063492064,
|
||||
"acc_norm": 0.40476190476190477
|
||||
},
|
||||
"hendrycksTest-marketing": {
|
||||
"acc": 0.6965811965811965,
|
||||
"acc_norm": 0.6837606837606838
|
||||
},
|
||||
"hendrycksTest-college_mathematics": {
|
||||
"acc": 0.23,
|
||||
"acc_norm": 0.29
|
||||
},
|
||||
"hendrycksTest-electrical_engineering": {
|
||||
"acc": 0.4827586206896552,
|
||||
"acc_norm": 0.4
|
||||
},
|
||||
"hendrycksTest-high_school_government_and_politics": {
|
||||
"acc": 0.5284974093264249,
|
||||
"acc_norm": 0.5492227979274611
|
||||
},
|
||||
"hendrycksTest-virology": {
|
||||
"acc": 0.37349397590361444,
|
||||
"acc_norm": 0.39759036144578314
|
||||
},
|
||||
"hendrycksTest-high_school_mathematics": {
|
||||
"acc": 0.2518518518518518,
|
||||
"acc_norm": 0.28888888888888886
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"hendrycksTest-high_school_european_history": 0,
|
||||
"hendrycksTest-college_chemistry": 0,
|
||||
"hendrycksTest-international_law": 0,
|
||||
"hendrycksTest-high_school_macroeconomics": 0,
|
||||
"hendrycksTest-miscellaneous": 0,
|
||||
"hendrycksTest-professional_law": 0,
|
||||
"hendrycksTest-medical_genetics": 0,
|
||||
"hendrycksTest-high_school_world_history": 0,
|
||||
"hendrycksTest-professional_medicine": 0,
|
||||
"hendrycksTest-moral_disputes": 0,
|
||||
"hendrycksTest-high_school_geography": 0,
|
||||
"hendrycksTest-high_school_microeconomics": 0,
|
||||
"hendrycksTest-machine_learning": 0,
|
||||
"hendrycksTest-security_studies": 0,
|
||||
"hendrycksTest-world_religions": 0,
|
||||
"hendrycksTest-conceptual_physics": 0,
|
||||
"hendrycksTest-high_school_physics": 0,
|
||||
"hendrycksTest-nutrition": 0,
|
||||
"hendrycksTest-high_school_psychology": 0,
|
||||
"hendrycksTest-professional_accounting": 0,
|
||||
"hendrycksTest-human_aging": 0,
|
||||
"hendrycksTest-college_physics": 0,
|
||||
"hendrycksTest-high_school_chemistry": 0,
|
||||
"hendrycksTest-high_school_biology": 0,
|
||||
"hendrycksTest-us_foreign_policy": 0,
|
||||
"hendrycksTest-philosophy": 0,
|
||||
"hendrycksTest-logical_fallacies": 0,
|
||||
"hendrycksTest-anatomy": 0,
|
||||
"hendrycksTest-jurisprudence": 0,
|
||||
"hendrycksTest-high_school_computer_science": 0,
|
||||
"hendrycksTest-elementary_mathematics": 0,
|
||||
"hendrycksTest-abstract_algebra": 0,
|
||||
"hendrycksTest-prehistory": 0,
|
||||
"hendrycksTest-moral_scenarios": 0,
|
||||
"hendrycksTest-college_medicine": 0,
|
||||
"hendrycksTest-econometrics": 0,
|
||||
"hendrycksTest-human_sexuality": 0,
|
||||
"hendrycksTest-management": 0,
|
||||
"hendrycksTest-computer_security": 0,
|
||||
"hendrycksTest-college_computer_science": 0,
|
||||
"hendrycksTest-public_relations": 0,
|
||||
"hendrycksTest-sociology": 0,
|
||||
"hendrycksTest-global_facts": 0,
|
||||
"hendrycksTest-astronomy": 0,
|
||||
"hendrycksTest-high_school_statistics": 0,
|
||||
"hendrycksTest-professional_psychology": 0,
|
||||
"hendrycksTest-high_school_us_history": 0,
|
||||
"hendrycksTest-business_ethics": 0,
|
||||
"hendrycksTest-clinical_knowledge": 0,
|
||||
"hendrycksTest-college_biology": 0,
|
||||
"hendrycksTest-formal_logic": 0,
|
||||
"hendrycksTest-marketing": 0,
|
||||
"hendrycksTest-college_mathematics": 0,
|
||||
"hendrycksTest-electrical_engineering": 0,
|
||||
"hendrycksTest-high_school_government_and_politics": 0,
|
||||
"hendrycksTest-virology": 0,
|
||||
"hendrycksTest-high_school_mathematics": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-7b-v2",
|
||||
"num_fewshot": 5,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"results": {
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.2423500611995104,
|
||||
"mc2": 0.3645856452722409
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"truthfulqa_mc": 1
|
||||
},
|
||||
"config": {
|
||||
"model": "stabilityai/stablelm-base-alpha-7b-v2",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": 8,
|
||||
"device": "cuda:0",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 10000,
|
||||
"description_dict": null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.3242320819112628,
|
||||
"acc_stderr": 0.013678810399518815,
|
||||
"acc_norm": 0.3506825938566553,
|
||||
"acc_norm_stderr": 0.013944635930726085
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.6725589225589226,
|
||||
"acc_stderr": 0.009629415859100609,
|
||||
"acc_norm": 0.6325757575757576,
|
||||
"acc_norm_stderr": 0.009892552616211548
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.645565749235474,
|
||||
"acc_stderr": 0.008366245832688784
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5105556662019518,
|
||||
"acc_stderr": 0.004988669343786956,
|
||||
"acc_norm": 0.6858195578570006,
|
||||
"acc_norm_stderr": 0.004632399677490817
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 3.995184075578421,
|
||||
"ppl_stderr": 0.08502310534019014,
|
||||
"acc": 0.7025033960799534,
|
||||
"acc_stderr": 0.006369088639380684
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.264,
|
||||
"acc_stderr": 0.019732885585922098,
|
||||
"acc_norm": 0.386,
|
||||
"acc_norm_stderr": 0.021793529219281165
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7600652883569097,
|
||||
"acc_stderr": 0.009963625892809545,
|
||||
"acc_norm": 0.780195865070729,
|
||||
"acc_norm_stderr": 0.00966195861665176
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.921,
|
||||
"acc_stderr": 0.008534156773333438,
|
||||
"acc_norm": 0.868,
|
||||
"acc_norm_stderr": 0.01070937396352803
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4247697031729785,
|
||||
"acc_stderr": 0.011185271257671346,
|
||||
"acc_norm": 0.4600818833162743,
|
||||
"acc_norm_stderr": 0.011277955967920396
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.22399020807833536,
|
||||
"mc1_stderr": 0.014594964329474202,
|
||||
"mc2": 0.35868737415331753,
|
||||
"mc2_stderr": 0.013670666454421172
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6211523283346487,
|
||||
"acc_stderr": 0.013633724603180328
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=stabilityai/stablelm-base-alpha-3b-v2,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:2",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.2363481228668942,
|
||||
"acc_stderr": 0.012414960524301829,
|
||||
"acc_norm": 0.257679180887372,
|
||||
"acc_norm_stderr": 0.012780770562768407
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.4473905723905724,
|
||||
"acc_stderr": 0.010202832385415646,
|
||||
"acc_norm": 0.42045454545454547,
|
||||
"acc_norm_stderr": 0.010129114278546535
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.5764525993883792,
|
||||
"acc_stderr": 0.008642220663071512
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.329416450906194,
|
||||
"acc_stderr": 0.004690407826933909,
|
||||
"acc_norm": 0.38309101772555265,
|
||||
"acc_norm_stderr": 0.004851466623601455
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 20.187359473367042,
|
||||
"ppl_stderr": 0.7391414436494796,
|
||||
"acc": 0.4172326799922375,
|
||||
"acc_stderr": 0.006869874864639983
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.17,
|
||||
"acc_stderr": 0.016815633531393422,
|
||||
"acc_norm": 0.294,
|
||||
"acc_norm_stderr": 0.020395095484936614
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.717,
|
||||
"acc_stderr": 0.01425181090648174,
|
||||
"acc_norm": 0.649,
|
||||
"acc_norm_stderr": 0.015100563798316403
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.3561924257932446,
|
||||
"acc_stderr": 0.010836006561369118,
|
||||
"acc_norm": 0.4094165813715456,
|
||||
"acc_norm_stderr": 0.01112684957658903
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.6381936887921654,
|
||||
"acc_stderr": 0.011211397313020371,
|
||||
"acc_norm": 0.6273122959738846,
|
||||
"acc_norm_stderr": 0.01128131833289774
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.22399020807833536,
|
||||
"mc1_stderr": 0.014594964329474205,
|
||||
"mc2": 0.4052844601694033,
|
||||
"mc2_stderr": 0.014547007787950397
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.526440410418311,
|
||||
"acc_stderr": 0.014032823874407229
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=stabilityai/stablelm-base-alpha-3b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "8",
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.3848122866894198,
|
||||
"acc_stderr": 0.014218371065251107,
|
||||
"acc_norm": 0.4052901023890785,
|
||||
"acc_norm_stderr": 0.014346869060229334
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.7319023569023569,
|
||||
"acc_stderr": 0.009089526578213694,
|
||||
"acc_norm": 0.6910774410774411,
|
||||
"acc_norm_stderr": 0.009481048387761353
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.7030581039755351,
|
||||
"acc_stderr": 0.007991418738281637
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.5553674566819359,
|
||||
"acc_stderr": 0.004959094146471525,
|
||||
"acc_norm": 0.7426807408882693,
|
||||
"acc_norm_stderr": 0.004362633637374482
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 3.366151815984524,
|
||||
"ppl_stderr": 0.06573496070440447,
|
||||
"acc": 0.7418979235396856,
|
||||
"acc_stderr": 0.006096490478492321
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.304,
|
||||
"acc_stderr": 0.020591649571224925,
|
||||
"acc_norm": 0.418,
|
||||
"acc_norm_stderr": 0.022080014812228134
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.7845484221980413,
|
||||
"acc_stderr": 0.009592463115658117,
|
||||
"acc_norm": 0.8019586507072906,
|
||||
"acc_norm_stderr": 0.009298209954776726
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.939,
|
||||
"acc_stderr": 0.007572076091557423,
|
||||
"acc_norm": 0.917,
|
||||
"acc_norm_stderr": 0.00872852720607479
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.4242579324462641,
|
||||
"acc_stderr": 0.011183502662341787,
|
||||
"acc_norm": 0.4692937563971341,
|
||||
"acc_norm_stderr": 0.011292714928103489
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.2423500611995104,
|
||||
"mc1_stderr": 0.01500067437357034,
|
||||
"mc2": 0.36457253410937596,
|
||||
"mc2_stderr": 0.0135567172376818
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.6882399368587214,
|
||||
"acc_stderr": 0.013018571197638537
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=stabilityai/stablelm-base-alpha-7b-v2,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "2",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:2",
|
||||
"no_cache": false,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"results": {
|
||||
"arc_challenge": {
|
||||
"acc": 0.24829351535836178,
|
||||
"acc_stderr": 0.012624912868089755,
|
||||
"acc_norm": 0.27047781569965873,
|
||||
"acc_norm_stderr": 0.012980954547659556
|
||||
},
|
||||
"arc_easy": {
|
||||
"acc": 0.5054713804713805,
|
||||
"acc_stderr": 0.01025916922861504,
|
||||
"acc_norm": 0.44865319865319864,
|
||||
"acc_norm_stderr": 0.010205540414612876
|
||||
},
|
||||
"boolq": {
|
||||
"acc": 0.600611620795107,
|
||||
"acc_stderr": 0.008566178448007831
|
||||
},
|
||||
"hellaswag": {
|
||||
"acc": 0.3467436765584545,
|
||||
"acc_stderr": 0.004749606196363328,
|
||||
"acc_norm": 0.41216889065923124,
|
||||
"acc_norm_stderr": 0.004912192800263316
|
||||
},
|
||||
"lambada_openai": {
|
||||
"ppl": 9.459674222745228,
|
||||
"ppl_stderr": 0.33183598007493614,
|
||||
"acc": 0.5511352610130021,
|
||||
"acc_stderr": 0.006929452414790843
|
||||
},
|
||||
"openbookqa": {
|
||||
"acc": 0.214,
|
||||
"acc_stderr": 0.018359797502387025,
|
||||
"acc_norm": 0.32,
|
||||
"acc_norm_stderr": 0.02088234048876181
|
||||
},
|
||||
"piqa": {
|
||||
"acc": 0.6675734494015234,
|
||||
"acc_stderr": 0.010991141557445587,
|
||||
"acc_norm": 0.6735582154515778,
|
||||
"acc_norm_stderr": 0.010940467046177304
|
||||
},
|
||||
"sciq": {
|
||||
"acc": 0.801,
|
||||
"acc_stderr": 0.012631649083099177,
|
||||
"acc_norm": 0.697,
|
||||
"acc_norm_stderr": 0.014539683710535257
|
||||
},
|
||||
"siqa": {
|
||||
"acc": 0.394575230296827,
|
||||
"acc_stderr": 0.011059713589720797,
|
||||
"acc_norm": 0.4140225179119754,
|
||||
"acc_norm_stderr": 0.011145545345176117
|
||||
},
|
||||
"truthfulqa_mc": {
|
||||
"mc1": 0.23745410036719705,
|
||||
"mc1_stderr": 0.014896277441041836,
|
||||
"mc2": 0.3995908363542637,
|
||||
"mc2_stderr": 0.014371652685680641
|
||||
},
|
||||
"winogrande": {
|
||||
"acc": 0.5011838989739542,
|
||||
"acc_stderr": 0.014052446290529012
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"arc_challenge": 0,
|
||||
"arc_easy": 0,
|
||||
"boolq": 1,
|
||||
"hellaswag": 0,
|
||||
"lambada_openai": 0,
|
||||
"openbookqa": 0,
|
||||
"piqa": 0,
|
||||
"sciq": 0,
|
||||
"siqa": 0,
|
||||
"truthfulqa_mc": 1,
|
||||
"winogrande": 0
|
||||
},
|
||||
"config": {
|
||||
"model": "gpt2",
|
||||
"model_args": "use_fast=True,pretrained=stabilityai/stablelm-base-alpha-7b,trust_remote_code=True,low_cpu_mem_usage=True,dtype=auto",
|
||||
"num_fewshot": 0,
|
||||
"batch_size": "2",
|
||||
"batch_sizes": [],
|
||||
"device": "cuda:1",
|
||||
"no_cache": true,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"description_dict": {}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user