Model Rankings (Correctness Rate)
| Model | Correct Matches | Total | Correctness Rate |
|---|---|---|---|
| gemini-2.0-flash-001 | 28 | 28 | 100.0% |
| gemma-3-27b-it | 28 | 28 | 100.0% |
| o4-mini | 28 | 28 | 100.0% |
| qwen3-14b | 28 | 28 | 100.0% |
| qwen3-30b-a3b | 28 | 28 | 100.0% |
| qwen3-32b | 27 | 28 | 96.4% |
| qwen3-8b | 27 | 28 | 96.4% |
| qwen3-235b-a22b | 26 | 28 | 92.9% |
| gemini-2.5-pro-preview-03-25 | 24 | 28 | 85.7% |
| gemma-3-12b-it | 24 | 28 | 85.7% |
| llama-3.1-nemotron-ultra-253b-v1 | 24 | 28 | 85.7% |
| gpt-4.1-mini | 24 | 28 | 85.7% |
| gpt-4o-mini | 24 | 28 | 85.7% |
| o4-mini-high | 24 | 28 | 85.7% |
| gemini-2.5-flash-preview | 23 | 28 | 82.1% |
| gpt-4.1 | 23 | 28 | 82.1% |
| deepseek-prover-v2 | 19 | 28 | 67.9% |
| llama-4-maverick | 19 | 28 | 67.9% |
| gpt-4.1-nano | 19 | 28 | 67.9% |
| deepseek-r1 | 18 | 28 | 64.3% |
| llama-4-scout | 18 | 28 | 64.3% |
| deepseek-chat-v3-0324 | 9 | 28 | 32.1% |
| phi-4-reasoning-plus | 0 | 28 | 0.0% |
| phi-4-reasoning | 0 | 28 | 0.0% |
| llama-3.3-nemotron-super-49b-v1 | 0 | 28 | 0.0% |
Correctness Rate = (Number of responses with correctly matching layers against ground truth across all queries and runs) / (Total possible responses) × 100%