eleuther eval readme

The goal here is to run the Eleuther Eval harness exactly in the same way as that used in the huggingface LLM Leaderboard.

The starting point is a .bin file trained by llm.c. We now have to export it to a huggingface model and then evaluate it.

To export the model, use export_hf.py. See its documentation up top. Eample usage, from this directory:

cd dev/eval
python export_hf.py --input model.bin --output output_dir

Where you point to your model .bin file, and huggingface files get written to output_dir. The script can optionally also upload to huggingface hub. One more post-processing that is advisable is to go into the output_dir, open up the config.json there and add one more entry into the json object:

"_attn_implementation": "flash_attention_2"

To use FlashAttention 2. We had trouble evaluating in bfloat16 without using FlashAttention 2 (the scores are much lower, and this was never fully resolved). This is a temporary hack/workaround.

Now that we have the model in huggingface format, we download the Eleuther Eval Harness repo and run it. Head over to the parent/root directory of the llm.c repo and:

git clone https://github.com/EleutherAI/lm-evaluation-harness/
cd lm-evaluation-harness
git checkout b281b0921b636bc36ad05c0b0b0763bd6dd43463
pip install -e .

And then run the run_eval.sh script:

./dev/eval/run_eval.sh output_dir result_dir

Where output_dir can either be local output dir (above), or a huggingface repo name.This will write eval json objects to ./lm-evaluation-harness/results/results_dir. It will print the results into console, e.g. for a 774M model we see:

----------------------------------------
arc_challenge_25shot.json      : 30.4608
gsm8k_5shot.json               : 0.1516
hellaswag_10shot.json          : 57.8072
mmlu_5shot.json                : 25.8682
truthfulqa_0shot.json          : 35.7830
winogrande_5shot.json          : 59.3528
----------------------------------------
Average Score                  : 34.9039

But you can additionally get these results later by running summarize_eval.py:

python dev/eval/summarize_eval.py lm-evaluation-harness/results/results_dir

The same information will be printed again.

For some reason, the evaluation is quite expensive and runs for somewhere around 1-3 hours, even though it should be a few minutes at most. This has not been satisfyingly resolved so far.