The Department of War Small-Model Leaderboard: US models, one workstation GPU, zero network connections.
Independent, offline test results for US-made small language models on Department of War knowledge, honest staff work, and law-of-war judgment. The results come from AI Eval Corp's open-source evaluator, which is built to run air-gapped so a unit can rerun the tests on its own network.
As of October 10, 2026, LFM2.5-1.2B Instruct (1.17B) has the highest AIEval DoW Bench Score in v1.1.1 at 72.9%, with NVIDIA Nemotron 3 Nano 4B within half a point at 72.4%, among 9 US-developed small language models (1.17B to 5.12B total parameters) run fully offline on one RTX 4070 Super with zero network connections recorded. By suite, Nemotron leads DoW Knowledge (85.4%) and Honest Broker (58.3%), LFM2.5-2.6B leads Lawful Order (89.5%), and LFM2.5-1.2B Instruct and Phi-4-mini-instruct are the only models above 80% on both DoW Knowledge and Lawful Order. Honest Broker is the hardest suite (no model above 58.3%). Free-form items are graded by small local judges (Phi-4-mini-instruct for 7 models; Granite 4.2 3B for the 2 Phi-4-mini rows under the same-family rule), and the human-agreement check on that grading is still pending.
Decision-ready picks
Picks are computed from the published data and name the specific score in v1.1. AIEval DoW Bench Score: Equal-weight average of the three AI Eval curated benchmarks; weighting may change in a future version.
Results: AIEval DoW Bench Score and the three AI Eval curated benchmarks
DoW Knowledge, Honest Broker, and Lawful Order are AI Eval curated benchmarks, written by AI Eval Corp from public sources. The AIEval DoW Bench Score is their equal-weight average; weighting may change in a future version. Public-benchmark results are shown separately below and do not feed the score or ranking. Default order is by AIEval DoW Bench Score, highest first; click a column to sort. Each model name links to its run evidence.
| Model | Overall | AI Eval curated benchmarks | Sub-metrics | Run evidence | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Precision | ||||||||||||||||||||
Allen Institute for AI (Ai2) |
1.49B |
BF16 |
77.1% |
43.8% |
71.4% |
80.6% | 81.2% | 37.5% | 0.0% | 0.0% | 25.0% | 79.2% | 83.3% | 62.5% | 87.5% |
14.2 min853 s |
5,880 MiB5incl. ~2,641 MiB baseline |
|||
Liquid AI |
1.17B |
BF16 |
84.4% |
54.2% |
80.2% |
91.7% | 87.5% | 12.5% | 0.0% | 0.0% | 37.5% | 95.8% | 79.2% | 81.2% | 100.0% |
7.1 min423 s |
5,164 MiB5incl. ~2,645 MiB baseline |
|||
IBM |
1.46B |
BF16 |
83.3% |
29.2% |
72.9% |
90.3% | 75.0% | 37.5% | 0.0% | 0.0% | 25.0% | 50.0% | 87.5% | 62.5% | 87.5% |
21.6 min1,294 s |
5,872 MiB5incl. ~2,646 MiB baseline |
|||
Liquid AI |
2.70B |
BF16 |
56.2% |
42.5%3 |
89.5%3 |
58.3% | 68.8% | 12.5% | 12.5% | 12.5% | 37.5% | 65.2% | 91.7% | 87.5% | 100.0% |
95.4 min5,724 s |
11,482 MiB5incl. ~2,646 MiB baseline |
|||
IBM |
3.66B |
BF16 |
79.2% |
27.1% |
86.1% |
94.4% | 25.0% | 50.0% | 37.5% | 0.0% | 25.0% | 33.3% | 91.7% | 81.2% | 100.0% |
105.5 min6,327 s |
11,985 MiB5incl. ~2,588 MiB baseline |
|||
Microsoft |
3.84B |
BF16 |
80.2% |
16.7% |
80.8% |
94.4% | 37.5% | 37.5% | 0.0% | 37.5% | 12.5% | 16.7% | 87.5% | 75.0% | 50.0% |
13.4 min802 s |
8,149 MiB5incl. ~293 MiB baseline |
|||
Microsoft |
3.84B |
BF16 |
72.9% |
20.8% |
57.3% |
83.3% | 43.8% | 37.5% | 0.0% | 12.5% | 37.5% | 25.0% | 58.3% | 56.2% | 50.0% |
236.1 min14,168 s |
11,961 MiB5incl. ~302 MiB baseline |
|||
NVIDIA |
3.97B |
BF16 |
85.4% |
58.3% |
73.6% |
91.7% | 87.5% | 25.0% | 0.0% | 0.0% | 50.0% | 100.0% | 79.2% | 68.8% | 100.0% |
40.6 min2,435 s |
11,890 MiB5incl. ~1,650 MiB baseline |
|||
Google DeepMind |
5.12B total · 2.3B effective |
8-bit |
59.4% |
10.4% |
56.4% |
70.8% | 18.8% | 37.5% | 0.0% | 0.0% | 0.0% | 20.8% | 79.2% | 43.8% | 75.0% |
183.5 min11,012 s |
11,619 MiB5incl. ~337 MiB baseline |
|||
- AIEval DoW Bench Score: Equal-weight average of the three AI Eval curated benchmarks; weighting may change in a future version. Differences under about 1 point are not meaningful at these item counts: each model answers 192 items, and one Honest Broker item moves the overall score by about 0.7 points (one DoW Knowledge item by about 0.35). A model gets an overall score only when all three suite scores are complete; partial values are never averaged.
- Phi-4-mini-instruct and Phi-4-mini-reasoning: Graded by Granite 4.2 3B (same-family rule). These two rows use a different judge from the other 7 models, which are graded by Phi-4-mini-instruct. The judge never grades its own model family, so Granite 4.2 3B graded all 88 judge-graded items on each Phi row (0 over the judge window). Judge strictness differs, so free-form scores on these rows, especially Honest Broker, are not strictly comparable with the other 7. The human agreement check is pending.
- LFM2.5-2.6B is a reasoning model that always thinks. 52 of its 192 items hit the 2,048-token thinking cap, and 3 judge items were over the judge's 2,048-token window (1 Honest Broker rubric item, left out of that score; 2 Lawful Order lawful-task items, scored by the keyword/pattern check instead). Same settings as every model.
- Gemma 4 E2B-it ran at 8-bit (bitsandbytes LLM.int8). Google lists it at 11.4 GB in BF16, which leaves no room for context on a 12 GB card. Every other model ran at BF16. Its 5.12B total includes per-layer embeddings; Google markets it as 2.3B effective. Stage 1 counts total weights.
- Peak VRAM is whole-GPU memory during the run and includes a baseline from other processes on the workstation (shown under each value). All runs fit on one 12 GB RTX 4070 Super. Reasoning models get a 3,584-token context (2,048 thinking + 1,024 answer + 512 prompt); the others get 1,536.
Public benchmarks Indicative · sorted within group
Six widely used public benchmarks, run offline on the same workstation with the same llm-eval-suite kit. They are shown for context only and do not feed the AIEval DoW Bench Score or the ranking above. Each run used a subset smaller than the full benchmark, so scores are indicative and not directly comparable to published full-benchmark numbers.
| Model | Public benchmarks (percent correct) | ||||||
|---|---|---|---|---|---|---|---|
| Model and protocol | Avg (6) | MMLU-Pro | GSM8K | MATH-500 | IFEval (strict) | HumanEval+ | MBPP+ |
| Short subset (4 non-reasoning models) | |||||||
| Phi-4-mini-instruct Short subsetP1 | 72.5% | 58.0%n = 200 | 88.5%n = 200 | 65.0%n = 100 | 71.0%n = 100 | 70.5%n = 163 | 82.0%n = 50 |
| Granite 4.0 H 1B Short subsetP1 | 63.0% | 38.5%n = 200 | 68.5%n = 200 | 57.0%n = 100 | 73.0%n = 100 | 66.9%n = 163 | 74.0%n = 50 |
| LFM2.5-1.2B Instruct Short subsetP1 | 61.3% | 35.0%n = 200 | 70.0%n = 200 | 63.0%n = 100 | 78.0%n = 100 | 47.9%n = 163 | 74.0%n = 50 |
| OLMo 2 1B Instruct Short subsetP1 | 35.5% | 18.5%n = 200 | 66.5%n = 200 | 23.0%n = 100 | 59.0%n = 100 | 22.1%n = 163 | 24.0%n = 50 |
| Quick check (4 reasoning models, thinking capped) | |||||||
| NVIDIA Nemotron 3 Nano 4B Quick checkP2 | 73.3% | 62.0%n = 50 | 86.0%n = 50 | 66.0%n = 50 | 82.0%n = 50 | 84.0%n = 50 | 60.0%n = 50 |
| Granite 4.2 3B Quick checkP2 | 68.0% | 56.0%n = 50 | 90.0%n = 50 | 68.0%n = 50 | 60.0%n = 50 | 74.0%n = 50 | 60.0%n = 50 |
| LFM2.5-2.6B Quick checkP2 | 66.7% | 46.0%n = 50 | 80.0%n = 50 | 66.0%n = 50 | 86.0%n = 50 | 62.0%n = 50 | 60.0%n = 50 |
| Phi-4-mini-reasoning Quick checkP2 | 59.0% | 54.0%n = 50 | 92.0%n = 50 | 64.0%n = 50 | 46.0%n = 50 | 46.0%n = 50 | 52.0%n = 50 |
| Gemma 4 E2B-it Not runP3 | Not run: public benchmarks skipped at 8-bit (projected too slow) | ||||||
Models are sorted by the average of the six benchmarks within each group. The two groups used different protocols, so compare within a group, not across.
- P1 · Short subset (OLMo 2 1B, LFM2.5-1.2B, Granite 4.0 H 1B, Phi-4-mini-instruct): MMLU-Pro and GSM8K 200 items each, MATH-500 and IFEval 100, HumanEval+ 163 of 164, MBPP+ 50; no thinking budget; BF16. Coding scores use the corrected coding scorer (llm-eval-suite c4a1fa63).
- P2 · Quick check (LFM2.5-2.6B, Granite 4.2 3B, Phi-4-mini-reasoning, Nemotron 3 Nano 4B): these are reasoning models, so to keep run time practical they used a 50-item quick check per benchmark (seed 20261011) with thinking capped at 512 tokens; BF16. The capped thinking budget may lower their scores, and with 50 items one question moves a score by 2 points. Quick-check and short-subset scores are not directly comparable with each other. Coding scores were regraded on Oct 12, 2026 with the corrected coding scorer (llm-eval-suite d864b030), which scores the final answer rather than the thinking trace.
- P3 · Gemma 4 E2B-it: not run. At 8-bit (needed on a 12 GB card) the public-benchmark run was projected to take too long and was skipped.
- Avg (6): the plain mean of the six scores shown, used only to order models within a group. It is display-only: not part of the AIEval DoW Bench Score, the ranking, or the downloadable results files.
- Scorer: all public coding scores use the corrected coding scorer: llm-eval-suite d864b030 for the quick-check rows, and c4a1fa63 for the short-subset rows (the d864b030 change affects only reasoning output; OLMo 2 1B was re-checked under d864b030 with identical scores).
- Metrics: MMLU-Pro, GSM8K and MATH-500 are accuracy; IFEval is strict prompt-level accuracy; HumanEval+ and MBPP+ are pass@1 on the expanded “plus” tests.
- Reference exclusions: HumanEval/32 is excluded from HumanEval+, and MBPP+ tasks 255 and 596 are excluded, because the reference solutions do not pass their own tests reliably (upstream test data, and a platform-dependent result for 596).
- GPQA Diamond is not included. Its license does not allow redistributing the questions, so it is not part of the offline kit.
- Runs offline. The same kit, with these six benchmarks, installs and runs with no network connection, so a unit can reproduce these numbers on its own system. Air-gapped install guide.
Not ranked: Gemma 4 E4B and Arcee Trinity Nano are not ranked: marketed as smaller, but their total weights (≈8B; 6.1B) exceed the 6B cutoff.
What is not ranked and whyZero network connections
All 9 models ran fully offline on a single RTX 4070 Super, with every process in each run's process tree monitored. Every generation run and every judge pass, including the Nemotron rerun and the Granite 4.2 3B re-judge of the Phi rows, recorded 0 network entries, and the logs are summarized per run.
Our own open evaluator
llm-eval-suite is open source and has an offline mode and an air-gap install guide. Built to run fully air-gapped and designed for deployment on classified networks such as IL5, IL6, and JWICS-class enclaves; any installation requires that network's own accreditation.
Defense-specific tests that resist gaming
240 items from public sources. 216 stay private (the 192 scored held-out items plus a 24-item dev set), and the 24 published items carry a canary string. Judge grades are being checked against human grading, and the agreement rate will print next to the scores.
The same evaluator can run on your network
- Offline for this bench. The dow_bench preset turns on offline mode, loads weights only from local folders, and blocks cloud APIs. No third-party harness produced these scores.
- Reproducible. Every row carries a run ID, the evaluator commits for generation and scoring, model revision SHAs, the judge, and settings.
- Air-gap ready. Built to run fully air-gapped and designed for deployment on classified networks such as IL5, IL6, and JWICS-class enclaves; any installation requires that network's own accreditation.
- Name the task.
- Know your hardware.
- Know your network and data level.
- Check the license terms.
- Shortlist 2–3 from the table.
- Rerun them on your data with llm-eval-suite.
- Your ATO path is separate.
Led by Scott Mastin: Project Maven AI T&E program manager (2019–2023), Maven T&E lead under CDAO (2023–present), CDAO Rapid T&E Cell work under a Johns Hopkins APL subcontract, and Army Research Laboratory sensor and imagery AI testing (2007–2019).