wehub-resource-sync
917eedffcf
Main / Python 3.11 - Docs (push) Waiting to run
Main / Python 3.11 - Build (push) Waiting to run
Main / Python 3.11 - Lint (push) Waiting to run
Main / Python 3.11 - Style (push) Waiting to run
Main / Python 3.11 - Test (push) Waiting to run
Main / GPU CI (push) Blocked by required conditions
Main / Release (push) Blocked by required conditions
Main / Build and Push Docker Images (push) Blocked by required conditions
elo rating
Calculates elo rating of olmOCR vs other tools.
Data
The pairwise judgment data is stored in ratings.csv as win/loss counts:
MethodA,MethodB,A_wins,B_wins,A_rate(%),B_rate(%)
marker,mineru,53,26,67.1,32.9
mineru,pdelf,22,55,28.6,71.4
gotocr_format,marker,26,45,36.6,63.4
marker,pdelf,31,49,38.8,61.3
gotocr_format,pdelf,29,41,41.4,58.6
gotocr_format,mineru,38,37,50.7,49.3
Note pdfelf is olmOCR.
Usage
To calculate elo ratings, run the following command:
python calculate_elo_ratings.py ratings.csv --num-bootstrap 5000 --num-elo-sims 100 --confidence-level 95 --seed 123
It should print something like:
Bootstrapped Elo Ratings (95% CI):
--------------------------------------------------
pdelf 1813.0 ± 84.9 [1605.9, 1930.0]
mineru 1545.2 ± 99.7 [1336.7, 1714.1]
marker 1429.1 ± 100.7 [1267.6, 1645.5]
gotocr_format 1212.7 ± 82.0 [1097.3, 1408.3]
Pairwise Significance Tests:
--------------------------------------------------
gotocr_format vs marker Δ = -216.3 [-470.8, 135.0] p = 0.218
gotocr_format vs mineru Δ = -332.5 [-567.5, 19.3] p = 0.051
gotocr_format vs pdelf Δ = -600.3 [-826.1, -344.3] p = 0.000*
marker vs mineru Δ = -116.1 [-365.4, 246.5] p = 0.430
marker vs pdelf Δ = -383.9 [-610.6, -10.9] p = 0.044*
mineru vs pdelf Δ = -267.8 [-517.3, 104.0] p = 0.135
which is also already saved in results.txt.
To generate boxplots of elo ratings, run the following command:
python draw_boxplots.py results.txt boxplots.png
which should save boxplots as boxplots.png.