Evaluation
How a seat-by-seat approach would have done on Missouri’s legislative elections of 2018, 2020, 2022, 2024, using only what was known before each one. These are tests of method on past elections. They are not a record of this forecast, which is scored after the 2026 results are certified.
What was tested
Each of the four elections, 2018, 2020, 2022 and 2024, was forecast for both chambers using only information available before that election: earlier results, the district’s earlier statewide lean where one existed on the same map, and who was on the ballot. Only contested seats with one Democrat and one Republican are scored. The 2016 election is used for training only, and the 2016 Senate District 4 special election is never scored.
Three plain baselines were run in every year (A, B and C below). A fourth, P, reproduces this site’s own approach as closely as the rules allow, and only for 2024: it needs an earlier election on the same district map, and 2022 is the first. P changes or drops what could not be reproduced as of 2024, and so is not the 2026 model itself: the education term (its data was published later), the 2026 national shift (set to zero), the asymmetric incumbency adjustment, the primary-turnout and candidate-quality terms, the MO-2 feedback, and the 2024 composite lean (replaced by the 2020 presidential and governor results).
Which backtest this is
The Methodology page also shows a backtest, from data/forecast/backtest.json, which the hourly pipeline rewrites on every run. That one fits the production model on one election and forecasts the other (2022 and 2024), so the model has already seen data from the pair, and it is a retrospective diagnostic. The test on this page is newer in design and stricter: every statistic for a year uses only earlier elections, and a leakage guard in the code rejects any same-year input. Where the two differ, read this page for how well the method forecasts, and the Methodology table for how well the production model fits the two elections it was built from.
Were the intervals the right width?
For baseline A, the share of contested seats whose certified Democratic two-party share landed inside the model’s 50% interval (open dot) and 90% interval (filled dot). These are observed frequencies among seats, not probabilities of any one race. Dots on the dashed lines mean the intervals were the right width; dots to the right mean they were too wide.
Backtest run Nov 1, 2026, model A, contested seats only (n in each label). Seats in a year share one statewide environment, so they are not independent. The exact figures are in the table below.
Did the probabilities match what happened?
Baseline A’s win probabilities for all 427 contested seats of 2018 to 2024, grouped into ten bins. Across: the average predicted chance a Democrat wins in the bin. Up: the observed frequency, the share of seats in the bin that Democrats won. Bars are the Wilson 80% interval for that frequency. On the diagonal, the probabilities were right on average.
What it shows. At the ends, the baseline’s confident calls held. In the middle, seats went decisively one way more often than predicted: seats given 30% to 50% for a Democrat were won by Democrats 18% and 17% of the time, and seats given 60% to 70% were won 87% of the time. So even this plain baseline was, if anything, not confident enough in the middle.
What it does not show. This is baseline A, not the published forecast. The pattern sits in 2018 and 2020, on the old district map, and mostly in seats the baseline had no lean or past result for; on the current map (2022 and 2024) its middle seats were close to calibrated. The 2026 model is a different, tighter model whose seat-level error is measured out of sample, so this figure is not evidence that the 2026 chances are too cautious. The 2026 forecast will be added to this test after certification. The full analysis is in the decisions log (D-CAL1).
| Predicted chance a Democrat wins | Seats | Average predicted | Democrats won | Observed frequency | 80% interval |
|---|---|---|---|---|---|
| 0% to 10% | 207 | 2% | 0 | 0% | 0% to 1% |
| 10% to 20% | 33 | 14% | 2 | 6% | 3% to 14% |
| 20% to 30% | 25 | 23% | 4 | 16% | 9% to 27% |
| 30% to 40% | 17 | 35% | 3 | 18% | 9% to 32% |
| 40% to 50% | 48 | 43% | 8 | 17% | 11% to 25% |
| 50% to 60% | 11 | 53% | 6 | 55% | 36% to 72% |
| 60% to 70% | 15 | 64% | 13 | 87% | 72% to 94% |
| 70% to 80% (few seats) | 9 | 75% | 9 | 100% | 85% to 100% |
| 80% to 90% | 19 | 85% | 18 | 95% | 84% to 98% |
| 90% to 100% | 43 | 99% | 43 | 100% | 96% to 100% |
From data/backtest/calibration.json (model/backtest, Nov 1, 2026), baseline A, House and Senate pooled; download it on Data. Observed values are frequencies among seats. Bins with fewer than 10 seats are drawn open and should not be read alone. 4 election years; seats within a year share a statewide shift, so seats are not independent and the intervals are optimistic. The 2024 reproduction of this site’s approach is not shown here as a calibration claim: it is one election.
Scores by year, chamber and model
Brier score is the mean squared error of the win probabilities (0 is perfect; lower is better, and only scores for the same seats are comparable). Log loss punishes confident misses harder (probabilities clipped at 0.001). CRPS scores the whole distribution of the vote share (lower is better). Coverage is the observed frequency described above. Seat error is expected minus actual Democratic seats among the seats up (positive means the model gave Democrats too many).
| Chamber and year | Model | Seats | Brier | Log loss | CRPS | 50% cover | 90% cover | Seat error |
|---|---|---|---|---|---|---|---|---|
| House 2018 | A. District lean and incumbency | 117 | 0.069 | 0.238 | 0.059 | 48% | 91% | +11.1 |
| House 2018 | B. Last result, pulled toward lean | 117 | 0.072 | 0.252 | 0.058 | 54% | 90% | +12.2 |
| House 2018 | C. Uniform swing from last result | 117 | 0.069 | 0.238 | 0.059 | 48% | 91% | +11.0 |
| House 2020 | A. District lean and incumbency | 80 | 0.048 | 0.177 | 0.038 | 48% | 96% | +3.3 |
| House 2020 | B. Last result, pulled toward lean | 80 | 0.047 | 0.175 | 0.038 | 51% | 96% | +3.1 |
| House 2020 | C. Uniform swing from last result | 80 | 0.057 | 0.204 | 0.040 | 39% | 96% | +4.7 |
| House 2022 | A. District lean and incumbency | 69 | 0.074 | 0.248 | 0.022 | 93% | 100% | +1.3 |
| House 2022 | B. Last result, pulled toward lean | 69 | 0.075 | 0.249 | 0.022 | 93% | 100% | +1.3 |
| House 2022 | C. Uniform swing from last result | 69 | 0.075 | 0.248 | 0.022 | 93% | 100% | +1.2 |
| House 2024 | A. District lean and incumbency | 110 | 0.036 | 0.120 | 0.015 | 88% | 100% | −0.1 |
| House 2024 | B. Last result, pulled toward lean | 110 | 0.045 | 0.150 | 0.018 | 96% | 100% | +0.7 |
| House 2024 | C. Uniform swing from last result | 110 | 0.033 | 0.112 | 0.016 | 94% | 100% | +0.7 |
| House 2024 | P. This site's approach, reproduced (2024 only) | 110 | 0.026 | 0.092 | 0.012 | 80% | 100% | −0.7 |
| Senate 2018 | A. District lean and incumbency | 16 | 0.102 | 0.337 | 0.086 | 50% | 75% | +2.8 |
| Senate 2018 | B. Last result, pulled toward lean | 16 | 0.102 | 0.338 | 0.086 | 50% | 75% | +2.8 |
| Senate 2018 | C. Uniform swing from last result | 16 | 0.102 | 0.337 | 0.086 | 50% | 75% | +2.8 |
| Senate 2020 | A. District lean and incumbency | 10 | 0.233 | 0.650 | 0.119 | 30% | 60% | −1.6 |
| Senate 2020 | B. Last result, pulled toward lean | 10 | 0.226 | 0.627 | 0.119 | 30% | 70% | −1.6 |
| Senate 2020 | C. Uniform swing from last result | 10 | 0.253 | 0.723 | 0.119 | 30% | 60% | −1.5 |
| Senate 2022 | A. District lean and incumbency | 11 | 0.037 | 0.145 | 0.022 | 91% | 100% | +0.8 |
| Senate 2022 | B. Last result, pulled toward lean | 11 | 0.037 | 0.146 | 0.022 | 91% | 100% | +0.8 |
| Senate 2022 | C. Uniform swing from last result | 11 | 0.037 | 0.146 | 0.022 | 91% | 100% | +0.8 |
| Senate 2024 | A. District lean and incumbency | 14 | 0.017 | 0.089 | 0.013 | 93% | 100% | −0.2 |
| Senate 2024 | B. Last result, pulled toward lean | 14 | 0.017 | 0.091 | 0.013 | 93% | 100% | −0.2 |
| Senate 2024 | C. Uniform swing from last result | 14 | 0.023 | 0.105 | 0.014 | 86% | 100% | −0.4 |
| Senate 2024 | P. This site's approach, reproduced (2024 only) | 14 | 0.025 | 0.114 | 0.013 | 93% | 100% | −0.2 |
From data/backtest/scores.json (Nov 1, 2026). Frequencies are shares of contested seats.
All four elections together
| Chamber | Model | Years | Seats | Brier | Log loss | CRPS |
|---|---|---|---|---|---|---|
| House | A. District lean and incumbency | 2018, 2020, 2022, 2024 | 376 | 0.056 | 0.192 | 0.035 |
| House | B. Last result, pulled toward lean | 2018, 2020, 2022, 2024 | 376 | 0.059 | 0.205 | 0.035 |
| House | C. Uniform swing from last result | 2018, 2020, 2022, 2024 | 376 | 0.057 | 0.196 | 0.035 |
| House | P. This site's approach, reproduced (2024 only) | 2024 | 110 | 0.026 | 0.092 | 0.012 |
| Senate | A. District lean and incumbency | 2018, 2020, 2022, 2024 | 51 | 0.090 | 0.289 | 0.058 |
| Senate | B. Last result, pulled toward lean | 2018, 2020, 2022, 2024 | 51 | 0.089 | 0.285 | 0.058 |
| Senate | C. Uniform swing from last result | 2018, 2020, 2022, 2024 | 51 | 0.096 | 0.308 | 0.059 |
| Senate | P. This site's approach, reproduced (2024 only) | 2024 | 14 | 0.025 | 0.114 | 0.013 |
Pooled over contested seats. Model P covers 2024 only, so its row is not comparable with the four-election rows.
What these results do not show
- Few elections. Four test years are four observations of the statewide environment. No significance is claimed, and the House and Senate share each year’s environment, so they are not independent evidence.
- Differences among A, B and C are small and inconsistent from year to year.
- 2022 is entirely declared priors: there was no earlier election on the 2022 map to fit, so its scores say nothing about fitting, and its wide intervals cover nearly everything.
- Model P is one election, 2024. It had the best Brier score in the 2024 House (0.026, against 0.033 to 0.045 for A, B and C), but in the 2024 Senate it scored 0.025 against 0.017 for A and for B. Its carry-forward weights and error structure were tuned by the author on the 2022 and 2024 data, so that is not evidence of skill in 2026.
- The Senate has 10 to 16 contested races a year, 51 in all. Its scores are very noisy.
- Thin inputs on the old map. There is no statewide lean before 2022, so 2018 and 2020 rest on earlier legislative results, which exist only for seats that were contested before. The 2020 lean on the 2022 map was computed in 2026 from 2020 precinct results; the 2022 map was adopted before the 2022 election, so it was knowable, but split-precinct uncertainty is not carried.
- Uncontested seats are treated as certain, and third-party or write-in outcomes are absent from this data.
- Calibration rests on 427 seats in four elections. Seats in a year share one statewide shift, so they are not independent and the intervals in the calibration figure are optimistic.
From docs/planning/2026-11-01-accuracy-impl/E-backtest.md, “Honest limits”.
The rule for 2026, written before the vote
Scored after certification: Brier score and log loss over contested seats, seats called correctly, calibration by probability bin, and whether the chamber seat counts fall inside the 10th-90th percentile range. Compared against State Navigate (Nov 2), On Point Politics (Nov 1) and a lean-only baseline.
From the forecast file’s scoring field. The forecast is re-run every hour. The version published before polls close is saved under the git tag preclose-2026 and is the one scored. The scorecard fills in after certification. Every decision behind the model is on Decisions.