wehub-resource-sync
a789495a98
FreeBSD Smoke / FreeBSD Smoke (x86_64) (push) Has been cancelled
CI / Quality Guardrails (push) Has been cancelled
CI / Build & Test (macos-latest) (push) Has been cancelled
CI / Build & Test (ubuntu-latest) (push) Has been cancelled
CI / Build & Test (windows-latest) (push) Has been cancelled
CI / Format (push) Has been cancelled
CI / PowerShell Syntax (push) Has been cancelled
CI / Windows Cross-Target Check (Linux) (push) Has been cancelled
2.6 KiB
2.6 KiB
| 1 | # Claude Code + Opus 4.8 baseline on Terminal-Bench (5 trials/task). jcode must never do worse. | |||
|---|---|---|---|---|
| 2 | # task | trials | resolved | rate |
| 3 | adaptive-rejection-sampler | 5 | 4 | 80.0 |
| 4 | bn-fit-modify | 5 | 5 | 100.0 |
| 5 | break-filter-js-from-html | 5 | 5 | 100.0 |
| 6 | build-cython-ext | 5 | 5 | 100.0 |
| 7 | build-pmars | 5 | 5 | 100.0 |
| 8 | build-pov-ray | 5 | 5 | 100.0 |
| 9 | caffe-cifar-10 | 5 | 5 | 100.0 |
| 10 | cancel-async-tasks | 5 | 1 | 20.0 |
| 11 | chess-best-move | 5 | 2 | 40.0 |
| 12 | circuit-fibsqrt | 5 | 5 | 100.0 |
| 13 | cobol-modernization | 5 | 5 | 100.0 |
| 14 | code-from-image | 5 | 5 | 100.0 |
| 15 | compile-compcert | 5 | 3 | 60.0 |
| 16 | configure-git-webserver | 5 | 1 | 20.0 |
| 17 | constraints-scheduling | 5 | 5 | 100.0 |
| 18 | count-dataset-tokens | 5 | 5 | 100.0 |
| 19 | crack-7z-hash | 5 | 5 | 100.0 |
| 20 | custom-memory-heap-crash | 5 | 5 | 100.0 |
| 21 | db-wal-recovery | 5 | 0 | 0.0 |
| 22 | distribution-search | 5 | 5 | 100.0 |
| 23 | dna-assembly | 5 | 3 | 60.0 |
| 24 | dna-insert | 5 | 0 | 0.0 |
| 25 | extract-elf | 5 | 3 | 60.0 |
| 26 | extract-moves-from-video | 5 | 0 | 0.0 |
| 27 | feal-differential-cryptanalysis | 5 | 5 | 100.0 |
| 28 | feal-linear-cryptanalysis | 5 | 5 | 100.0 |
| 29 | filter-js-from-html | 5 | 3 | 60.0 |
| 30 | financial-document-processor | 5 | 2 | 40.0 |
| 31 | fix-code-vulnerability | 5 | 5 | 100.0 |
| 32 | fix-git | 5 | 5 | 100.0 |
| 33 | fix-ocaml-gc | 5 | 5 | 100.0 |
| 34 | gcode-to-text | 5 | 5 | 100.0 |
| 35 | git-leak-recovery | 5 | 5 | 100.0 |
| 36 | git-multibranch | 5 | 5 | 100.0 |
| 37 | gpt2-codegolf | 5 | 4 | 80.0 |
| 38 | headless-terminal | 5 | 5 | 100.0 |
| 39 | hf-model-inference | 5 | 5 | 100.0 |
| 40 | install-windows-3.11 | 5 | 4 | 80.0 |
| 41 | kv-store-grpc | 5 | 5 | 100.0 |
| 42 | large-scale-text-editing | 5 | 5 | 100.0 |
| 43 | largest-eigenval | 5 | 4 | 80.0 |
| 44 | llm-inference-batching-scheduler | 5 | 5 | 100.0 |
| 45 | log-summary-date-ranges | 5 | 5 | 100.0 |
| 46 | mailman | 5 | 5 | 100.0 |
| 47 | make-doom-for-mips | 5 | 0 | 0.0 |
| 48 | make-mips-interpreter | 5 | 4 | 80.0 |
| 49 | mcmc-sampling-stan | 5 | 5 | 100.0 |
| 50 | merge-diff-arc-agi-task | 5 | 5 | 100.0 |
| 51 | model-extraction-relu-logits | 5 | 0 | 0.0 |
| 52 | modernize-scientific-stack | 5 | 5 | 100.0 |
| 53 | mteb-leaderboard | 5 | 5 | 100.0 |
| 54 | mteb-retrieve | 5 | 2 | 40.0 |
| 55 | multi-source-data-merger | 5 | 5 | 100.0 |
| 56 | nginx-request-logging | 5 | 5 | 100.0 |
| 57 | openssl-selfsigned-cert | 5 | 4 | 80.0 |
| 58 | overfull-hbox | 5 | 4 | 80.0 |
| 59 | password-recovery | 5 | 5 | 100.0 |
| 60 | path-tracing | 5 | 4 | 80.0 |
| 61 | path-tracing-reverse | 5 | 4 | 80.0 |
| 62 | polyglot-c-py | 5 | 5 | 100.0 |
| 63 | polyglot-rust-c | 5 | 5 | 100.0 |
| 64 | portfolio-optimization | 5 | 4 | 80.0 |
| 65 | protein-assembly | 5 | 0 | 0.0 |
| 66 | prove-plus-comm | 5 | 5 | 100.0 |
| 67 | pypi-server | 5 | 5 | 100.0 |
| 68 | pytorch-model-cli | 5 | 2 | 40.0 |
| 69 | pytorch-model-recovery | 5 | 3 | 60.0 |
| 70 | qemu-alpine-ssh | 5 | 4 | 80.0 |
| 71 | qemu-startup | 5 | 5 | 100.0 |
| 72 | query-optimize | 5 | 5 | 100.0 |
| 73 | raman-fitting | 5 | 0 | 0.0 |
| 74 | regex-chess | 5 | 5 | 100.0 |
| 75 | regex-log | 5 | 5 | 100.0 |
| 76 | reshard-c4-data | 5 | 5 | 100.0 |
| 77 | rstan-to-pystan | 5 | 1 | 20.0 |
| 78 | sam-cell-seg | 5 | 4 | 80.0 |
| 79 | sanitize-git-repo | 5 | 4 | 80.0 |
| 80 | schemelike-metacircular-eval | 5 | 5 | 100.0 |
| 81 | sparql-university | 5 | 5 | 100.0 |
| 82 | sqlite-db-truncate | 5 | 5 | 100.0 |
| 83 | sqlite-with-gcov | 5 | 5 | 100.0 |
| 84 | torch-pipeline-parallelism | 5 | 4 | 80.0 |
| 85 | torch-tensor-parallelism | 5 | 5 | 100.0 |
| 86 | train-fasttext | 5 | 0 | 0.0 |
| 87 | tune-mjcf | 5 | 5 | 100.0 |
| 88 | video-processing | 5 | 0 | 0.0 |
| 89 | vulnerable-secret | 5 | 5 | 100.0 |
| 90 | winning-avg-corewars | 5 | 5 | 100.0 |
| 91 | write-compressor | 5 | 4 | 80.0 |