Methodology
How AI Eval Corp ran stage 1: what was eligible, how the runs stayed offline, how items are scored, and what the numbers cannot tell you.
Eligibility
- US developers only. The developer must be US-based. Models from non-US developers are not eligible, however capable or easy to run locally.
- 6B and under by total weights. A smaller “effective” or “active” count does not bring a larger model in. See Excluded models.
- License must not prohibit military use. Each model's terms are badged and summarized on its page. Published license terms do not prohibit military use (quoted and linked). Not legal advice; confirm with your contracting officer or SJA.
- Open weights, so the run can happen on a closed network.
The offline run
All 9 models ran on one NVIDIA RTX 4070 Super (12 GB) on a Windows workstation, Oct 8–9, 2026 (ET). Eight ran with llm-eval-suite at commit 42532423; NVIDIA Nemotron 3 Nano 4B was rerun on Oct 9 at 32c27a23 after an end-token fix (the first run padded every answer to the 1,024-token cap), and its judged scores were identical. Weights were downloaded beforehand at pinned revision SHAs, verified, and loaded from local folders with offline mode on (LLM_EVAL_OFFLINE=1) and remote code off.
Network monitoring. A monitor watched the full process tree of every run (launcher and interpreter, not only the launcher) and logged TCP/UDP entries and netstat lines. Every generation run, its sanity check, and every judge pass recorded 0 entries. Final rescoring at 32c27a23 (and c52abb57 for the two re-judged Phi rows) reads the saved outputs and loads no model. GPU memory was sampled with nvidia-smi once per second.
Settings
Every model gets the same answer cap (1,024 tokens) and prompt budget (512 tokens). Models with a thinking mode (LFM2.5-2.6B, Granite 4.2 3B, Nemotron 3 Nano 4B, Phi-4-mini-reasoning) get a separate 2,048-token thinking cap; if a think block is still open at the cap, it is closed and the answer phase continues. Precision rule: full precision (BF16) where the model fits with room for context; otherwise 8-bit. Only Gemma 4 E2B ran at 8-bit.
| Model | Precision | Thinking cap | Answer cap | Context | Run ID | Judge | Net entries (run) | Net entries (judge) |
|---|---|---|---|---|---|---|---|---|
| OLMo 2 1B Instruct | BF16 | — | 1024 | 1536 | 441f970d4d5a | Phi-4-mini-instruct | 0 | 0 |
| LFM2.5-1.2B Instruct | BF16 | — | 1024 | 1536 | 307121353604 | Phi-4-mini-instruct | 0 | 0 |
| Granite 4.0 H 1B | BF16 | — | 1024 | 1536 | ad10318e45fb | Phi-4-mini-instruct | 0 | 0 |
| LFM2.5-2.6B | BF16 | 2048 | 1024 | 3584 | e636eb6b719f | Phi-4-mini-instruct | 0 | 0 |
| Granite 4.2 3B | BF16 | 2048 | 1024 | 3584 | 64fe6c3e6959 | Phi-4-mini-instruct | 0 | 0 |
| Phi-4-mini-instruct | BF16 | — | 1024 | 1536 | 9bf1f0609013 | Granite 4.2 3B | 0 | 0 |
| Phi-4-mini-reasoning | BF16 | 2048 | 1024 | 3584 | ff5dfd0efd0e | Granite 4.2 3B | 0 | 0 |
| NVIDIA Nemotron 3 Nano 4B | BF16 | 2048 | 1024 | 3584 | 7fef66483e47 | Phi-4-mini-instruct | 0 | 0 |
| Gemma 4 E2B-it | 8-bit | — | 1024 | 1536 | 8bc3e0fc8619 | Phi-4-mini-instruct | 0 | 0 |
Net entries (judge) for the two Phi rows are from the Granite 4.2 3B re-judge.
The three AI Eval curated benchmarks
All three suites are AI Eval curated benchmarks: written by AI Eval Corp from public sources, tagged to a source paragraph, and written to describe intent rather than ask for technical weapon detail. Each model answered the same 192 held-out items.
- DoW Knowledge (96 items). Department of War issuances, doctrine, and policy: 72 multiple-choice, 16 short-answer (judge-graded), and 8 name-the-issuance items. Score = share of items passed.
- Honest Broker (48 items). Honest staff work: abstaining on fake issuances (8), citation validity (8), sycophancy pairs (8), and judge-graded rubric items (24). Score = share of items passed.
- Lawful Order (48 items). Law-of-war judgment: harmonic mean of refusal-correctness on 24 unlawful requests and helpfulness on 16 lawful-but-uncomfortable tasks, so refusing everything cannot win. 8 gray-area items are reported separately.
The AIEval DoW Bench Score
The AIEval DoW Bench Score is the equal-weight average of the three suite headline scores: (DoW Knowledge + Honest Broker + Lawful Order) / 3. Weighting may change in a future version. A model gets an overall score only when all three suite scores are complete; a model missing any suite score would get no overall score and no rank. All 9 models have complete scores in this update. Partial values are never averaged. Rank is by the unrounded average.
Differences under about 1 point are not meaningful at these item counts. Each model answers 192 items; one Honest Broker item moves the overall score by about 0.7 points and one DoW Knowledge item by about 0.35. Read the three suite scores alongside it: they measure different things, and the Lawful Order headline already balances refusing against helping.
The judge
Free-form items are graded by a local judge model run through the same suite with a 2,048-token window. Phi-4-mini-instruct judges the other 7 models. The judge never grades its own model family, so Granite 4.2 3B judges Phi-4-mini-instruct and Phi-4-mini-reasoning (the same-family rule). Granite 4.2 3B graded all 88 judge-graded items on each Phi row, with 0 over the judge window and 0 network entries, at llm-eval-suite c52abb57. Judge strictness differs, so free-form scores on the two Phi rows, especially Honest Broker, are not strictly comparable with the other 7. On the other rows, an item too long for the judge window is left out of a judged metric; on Lawful Order it falls back to the keyword/pattern check.
Human agreement: Pending human grading check (≥80% gate). A 30-item stratified sample across DoW Knowledge short answers, Honest Broker, and Lawful Order, covering both judges, is graded by a person, and the agreement rate is printed next to the scores. At 80% or better the free-form scores publish with that number; below 80% they are withheld and only deterministic scores go out.
Where a small judge is weak: it can grade fluency rather than whether the right principle was named, it is inconsistent on partial credit, and it shares blind spots with the models it grades. Citation validity and fake-issuance abstention are checked deterministically against a whitelist, with no judge involved.
Contamination controls
- 240 items from public sources. 192 are the held-out test set that produces every score, 24 are a dev set used only to tune the judge prompt (never scored), and 24 are a public sample published with a canary string.
- The 216 private items are kept out of public repos and out of any training data.
- Planned rotation: about 20% of items each quarter, with version tags, and we watch the gap between public-sample and private-test scores.
Known limits
- Small local judges (Phi-4-mini-instruct, 3.8B; Granite 4.2 3B for the two Phi rows), with human agreement still pending. Free-form scores from the two judges are not strictly comparable.
- A small item count: 192 per model. One item moves DoW Knowledge by about 1 point, Honest Broker by about 2, and the AIEval DoW Bench Score by under 1.
- One hardware profile (RTX 4070 Super, 12 GB) and one run per model. Tokens per second were not measured in v1.1.
- Peak VRAM is whole-GPU memory and includes other processes on the workstation.
- Public-benchmark results (added in v1.1) use smaller subsets than the full benchmarks, are indicative only, and do not feed the score or ranking. See Public benchmarks.
Versioning, corrections, and appeals
A permanent v1.1.1 snapshot URL is coming. It will carry the date, the llm-eval-suite commits, model revision SHAs, and data-file hashes. Old snapshots are never edited, only superseded. Model makers can email scott@aieval.org with the subject “Leaderboard correction”; we acknowledge within 2 business days, rerun on a shown configuration error, and add vendor-recommended settings as a labeled extra row. Every change goes in the public, dated changelog.