Model performance
Validation
Historical match forecasts compared with available betting odds, closing where available. Lower is better for both metrics; the market is a hard baseline — the goal is to stay close to it.
As of Sep 4, 2026
Quarterly frozen forecasts compared with available bookmaker odds (closing when available). Historical squad-value priors are excluded to avoid future information. This is not a test of the full production update cadence. See forecasts recorded before kickoff.
Season forecasts: a separate check
70 complete league-seasons. Title-position Brier: 0.0290 · Top-N: 0.0627 · Bottom-position: 0.0968. Lower is better.
17.3% of actual reconstructed finishes land in the outer predicted deciles; the reference is 20%. This diagnostic alone does not establish calibration.
Finish positions reconstructed from match points, GD and GF; bottom-position events are not official relegation outcomes. Deductions, historical qualification rules and playoff results are not modeled. Incomplete round-robin seasons are excluded.
Does wider uncertainty improve forecasts?
70 complete league-seasons, 1,378 team-seasons, checked at preseason and halfway through each schedule. These historical checks exclude current squad-value priors.
| Horizon / noise | Title Brier | Top-N Brier | Bottom Brier | Outer deciles |
|---|---|---|---|---|
| Preseason · 0 | 0.02896 | 0.06256 | 0.09808 | 23.7% |
| Preseason · 0.12 | 0.02905 | 0.06270 | 0.09680 | 17.3% |
| Midseason · 0 | 0.01489 | 0.03338 | 0.05579 | 21.3% |
| Midseason · 0.12 | 0.01514 | 0.03382 | 0.05581 | 17.5% |
The 0.12 setting widens distributions. It improves preseason bottom-position forecasts in this sample, but does not show a clear title or top-N benefit; midseason distributions appear too wide by this diagnostic. League-season bootstrap intervals include zero for all comparisons except preseason bottom positions. The production setting remains 0.12 pending stronger forward evidence.
Download paired comparisons and uncertainty intervalsForecast accuracy by season
| Window | Matches | RPS model | RPS market | Gap | Log loss model | Log loss market |
|---|---|---|---|---|---|---|
| 2024-25 (Aug-Oct) | 2,357 | 0.206 | 0.200 | +0.006 | 1.016 | 0.996 |
| 2024-25 (Nov-Jan) | 2,195 | 0.207 | 0.199 | +0.007 | 1.020 | 0.997 |
| 2024-25 (Feb-May) | 3,443 | 0.209 | 0.202 | +0.007 | 1.012 | 0.989 |
| 2025-26 (Aug-Oct) | 2,394 | 0.210 | 0.205 | +0.005 | 1.022 | 1.004 |
| 2025-26 (Nov-Jan) | 1,707 | 0.207 | 0.200 | +0.007 | 1.013 | 0.991 |
| 2025-26 (Feb-May) | 2,199 | 0.208 | 0.201 | +0.007 | 1.015 | 0.994 |
| vs-538 2022-23 (Aug-Oct) | 2,947 | 0.209 | 0.204 | +0.005 | 1.014 | 0.997 |
| vs-538 2022-23 (Nov-Feb) | 2,229 | 0.206 | 0.198 | +0.008 | 1.012 | 0.988 |
RPS is the ranked probability score over win/draw/loss. A positive gap means the betting market was sharper over that window.
Calibration
Forecast probabilities bucketed into bins: for each bin, how often did the predicted outcome actually happen? A well-calibrated model sits on the diagonal.
Hover a point for bin detail. Point size scales with the number of forecasts in the bin.
Data table
| Bin | Pred. | Obs. | n |
|---|---|---|---|
| 3% | 3.9% | 8.6% | 70 |
| 8% | 7.9% | 6.0% | 671 |
| 13% | 12.8% | 11.0% | 1,639 |
| 18% | 17.8% | 15.8% | 3,438 |
| 23% | 23.0% | 23.3% | 9,603 |
| 28% | 27.3% | 27.8% | 15,252 |
| 33% | 32.3% | 31.0% | 7,056 |
| 38% | 37.5% | 37.2% | 5,373 |
| 43% | 42.4% | 42.4% | 4,601 |
| 48% | 47.4% | 48.1% | 3,566 |
| 53% | 52.3% | 51.9% | 2,571 |
| 57% | 57.3% | 59.5% | 1,717 |
| 63% | 62.3% | 64.4% | 1,125 |
| 68% | 67.3% | 69.8% | 724 |
| 73% | 72.3% | 75.8% | 463 |
| 78% | 77.3% | 79.2% | 336 |
| 83% | 82.0% | 86.6% | 157 |
| 88% | 87.2% | 81.0% | 42 |
Accuracy by domestic league
| League | Matches | Model RPS | Market RPS |
|---|---|---|---|
| ARG1 | 970 | 0.2140 | 0.2099 |
| AUT1 | 347 | 0.2102 | 0.2104 |
| BEL1 | 820 | 0.2101 | 0.2045 |
| BRA1 | 640 | 0.2063 | 0.1950 |
| DNK1 | 340 | 0.2124 | 0.2062 |
| ENG1 | 982 | 0.2119 | 0.2014 |
| ENG2 | 1444 | 0.2189 | 0.2145 |
| ESP1 | 977 | 0.1999 | 0.1945 |
| ESP2 | 1183 | 0.2135 | 0.2062 |
| FRA1 | 849 | 0.2040 | 0.1979 |
| FRA2 | 827 | 0.2218 | 0.2129 |
| GER1 | 795 | 0.2036 | 0.1991 |
| GER2 | 758 | 0.2222 | 0.2165 |
| GRE1 | 618 | 0.1900 | 0.1848 |
| ITA1 | 989 | 0.1957 | 0.1900 |
| ITA2 | 979 | 0.2132 | 0.2052 |
| MEX1 | 606 | 0.2014 | 0.1951 |
| NED1 | 785 | 0.1934 | 0.1887 |
| NOR1 | 366 | 0.2077 | 0.1977 |
| POL1 | 551 | 0.2153 | 0.2108 |
| POR1 | 768 | 0.1844 | 0.1784 |
| SCO1 | 590 | 0.1946 | 0.1913 |
| SUI1 | 362 | 0.2156 | 0.2129 |
| SWE1 | 403 | 0.2177 | 0.2085 |
| TUR1 | 793 | 0.2013 | 0.1877 |
| USA1 | 729 | 0.2214 | 0.2153 |