Leaderboard / Models / Phi-4-mini-reasoning
AIEval DoW Bench v1.1.1 · Last verified Oct 10, 2026
Phi-4-mini-reasoning
Graded by Granite 4.2 3B (same-family rule)MITMicrosoft3.84BBF16 (full precision)
Not an ATO, endorsement, or approval for any network. AI Eval Corp is an independent company. This leaderboard is not sponsored, endorsed, or approved by the Department of War or any government agency. A listing is not an ATO or an authorization for any network.
Graded by Granite 4.2 3B (same-family rule). The primary judge, Phi-4-mini-instruct, never grades its own model family, so Granite 4.2 3B graded all 88 judge-graded items on this model (0 over the judge window). The other 7 models are graded by Phi-4-mini-instruct. Judge strictness differs, so free-form scores here, especially Honest Broker, are not strictly comparable with the other 7. The human agreement check is pending.
Scores
AIEval DoW Bench Score
Equal-weight average of the three AI Eval curated benchmarks; weighting may change in a future version.
50.3%
Rank 8 of 9. (72.9% + 20.8% + 57.3%) / 3 = 0.5034. Differences under about 1 point are not meaningful at these item counts.
DoW Knowledge
AI Eval curated benchmark
72.9%
Rank 7 of 9
- Items
- 96 held-out (96 scored)
- Run ID
- ff5dfd0efd0e
- Precision
- BF16 (full precision)
- Judge
- Granite 4.2 3B (local; same-family rule, all 88 judge-graded items, 0 over the judge window)
- Judge agreement
- Pending human grading check (≥80% gate)
- Suite version
- stage1
Honest Broker
AI Eval curated benchmark
20.8%
Rank 7 of 9
- Items
- 48 held-out (48 scored)
- Run ID
- ff5dfd0efd0e
- Precision
- BF16 (full precision)
- Judge
- Granite 4.2 3B (local; same-family rule, all 88 judge-graded items, 0 over the judge window)
- Judge agreement
- Pending human grading check (≥80% gate)
- Suite version
- stage1
Lawful Order
AI Eval curated benchmark
57.3%
Rank 8 of 9
- Items
- 48 held-out (48 scored)
- Run ID
- ff5dfd0efd0e
- Precision
- BF16 (full precision)
- Judge
- Granite 4.2 3B (local; same-family rule, all 88 judge-graded items, 0 over the judge window)
- Judge agreement
- Pending human grading check (≥80% gate)
- Suite version
- stage1
Sub-metrics
| Suite | Metric | Graded by | Result |
|---|---|---|---|
| DoW Knowledge | Multiple choice | Deterministic | 83.3% (60 of 72) |
| DoW Knowledge | Short answer | Judge | 43.8% (7 of 16) |
| DoW Knowledge | Name the issuance | Deterministic | 37.5% (3 of 8) |
| Honest Broker | Fake-issuance abstention | Deterministic | 0.0% (0 of 8) |
| Honest Broker | Citation validity | Deterministic | 12.5% (1 of 8) |
| Honest Broker | Sycophancy | Deterministic | 37.5% (3 of 8) |
| Honest Broker | Rubric | Judge | 25.0% (6 of 24) |
| Lawful Order | Refused unlawful requests | Judge | 58.3% (14 of 24) |
| Lawful Order | Helped on lawful tasks | Judge | 56.2% (9 of 16) |
| Lawful Order | Gray area (not in headline) | Judge | 50.0% (4 of 8) |
Lawful Order headline = harmonic mean of “Refused unlawful requests” and “Helped on lawful tasks.” Gray area is reported but not in the headline. Counts are rates times the item counts shown, rounded.
Run evidence
| Run ID | ff5dfd0efd0e |
|---|---|
| Evaluator | llm-eval-suite: generation at 42532423, re-judged and rescored at c52abb57 |
| Judge | Granite 4.2 3B (same-family rule), 2,048-token window |
| Hardware | One NVIDIA RTX 4070 Super (12 GB), Windows workstation |
| Network entries | 0 across the run's process tree (sanity run: 0; judge pass: 0) |
| Run time (192 items) | 14,168 s (236.1 min) |
| Peak VRAM (whole GPU) | 11,961 MiB, including ~302 MiB baseline from other processes |
| Answers that hit the 1,024-token cap | 7 of 192 |
| Thinking truncated at 2,048 tokens | 15 of 192 |
| Tokens per second | Not measured |
| Empty answers | 0 |
| Prompts over budget | 0 |
Model and settings
| Developer | Microsoft (US) |
|---|---|
| Weights | microsoft/Phi-4-mini-reasoning |
| Revision SHA | 0e3b1e2d02ee478a3743abe3f629e9c0cb722e0a |
| Total parameters | 3.836B |
| Effective / active parameters | Same as total (dense) |
| Architecture | Dense transformer; reasoning model |
| Precision run | BF16 (full precision) |
| Answer cap | 1,024 tokens |
| Thinking cap | 2,048 tokens |
| Context window | 3,584 tokens (prompt budget 512) |
| Remote code | Off (trust_remote_code false) |
License in plain English
MIT
MIT. No military-use restriction found.
Verbatim clause: Pending: quoted and dated at the release license re-check. Source: model card.
Published license terms do not prohibit military use (quoted and linked). Not legal advice; confirm with your contracting officer or SJA.
Rerun this yourself with llm-eval-suite
python -m dow_bench run --model "Phi-4-mini-reasoning" \ --folder PATH_TO_LOCAL_WEIGHTS --precision bf16 \ --max-new-tokens 1024 --output dow_leaderboard.csv python -m dow_bench judge --run-dir runs/RUN_ID
Weights load from a local folder with offline mode on. The held-out items are private; the public sample runs the same way. Air-gap install guide.