AIEval DoW Bench v1.1.1 · Last verified Oct 10, 2026
Data
Download the results under CC BY 4.0: one row per model and suite (27 rows), with the AIEval DoW Bench Score, sub-metrics, the judge, and run evidence. Test items, prompts, and model outputs are not included.
- results.csv sha256 84198e891f134f3e…
- results.json sha256 e5f5812872bf8353…
- changelog.json sha256 b60530c78ae3625f…
Columns that need explaining
| overall_score / overall_rank | AIEval DoW Bench Score: equal-weight mean of the three suite scores, 0–1, repeated on each of a model's rows. Computed only when all three suite scores are complete. |
| judge_model / judge_same_family_fallback | The local judge that graded free-form items: Phi-4-mini-instruct, or Granite 4.2 3B (true) for the Phi-4-mini rows, because a judge never grades its own family. |
| evaluator_commit_generation / _scoring | llm-eval-suite commits for the generation run and for judging and rescoring. |
| score | Suite headline, 0–1. Blank when not graded. |
| score_status | complete for every row in this update. |
| n_items / n_scored | Held-out items in the suite / items that count toward the score. |
| precision | bf16 or 8bit. |
| judge_agreement | Blank: pending human grading check (≥80% gate). |
| peak_vram_mib_whole_gpu | Whole-GPU peak during the run (nvidia-smi, 1 s); includes baseline_vram_mib from other processes. |
| runtime_stats_status | final for every row. |
| stopped_on_newline_run | Reported for the Nemotron rerun only (blank elsewhere). |
| tokens_per_s | Not measured in v1.1.1 (blank). |
| net_entries_run / net_entries_judge | TCP/UDP entries seen across the process tree during generation / the judge pass. |
Snapshots
v1.1.1 snapshot URL: Coming. Snapshots are never edited, only superseded.
Cite
AI Eval Corp. AIEval DoW Bench v1.1.1, last verified Oct 10, 2026. https://aieval.org/leaderboard/