A forecast that grades itself

Publishing a probability is cheap. Publishing one that survives being checked is not. Beat the Streak posts a number for every likely MLB starter each morning — the chance they get at least one hit — and then, the following night, looks up what actually happened and refits itself against the answer.

The model matters less than the loop around it. But the model is where the useful ideas are, so start there.

Step one: a base rate that respects sample size

A hitter batting .310 over 40 at-bats is not a .310 hitter; he is a hitter about whom you know very little. So every input rate is first regressed toward the league mean by a fixed prior weight:

reg(rate, n, denom) = (rate·n + league_avg·denom) / (n + denom)

Batting average is regressed with a denominator of 60 at-bats, pitcher batting-average-against with 120. A hitter with 400 AB keeps most of his own number; one with 30 is pulled most of the way back to league average, which is the honest reading of thirty at-bats.

Step two: log5, for matchup

Now combine a regressed hitter against a regressed pitcher. The classic tool is Bill James's log5, which asks what happens when a batter of quality b faces a pitcher who allows q, in a league averaging L:

log5(b, q, L) =        b·q/L
                ─────────────────────────
                b·q/L + (1−b)(1−q)/(1−L)

The intuition: it converts both rates into odds relative to league average, multiplies them, and converts back. A good hitter against a good pitcher lands sensibly between the two, rather than at their arithmetic mean — which would badly overrate the hitter in the extreme cases.

Park factor scales the result, clamped to ±6% so Coors Field cannot do too much work.

Step three: at-bats, which is where the lineup card earns its place

All of that yields a per-at-bat hit probability. But the question is about the game, so it needs the number of chances — and that comes from where a hitter bats in the order:

exp_ab = 4.35   if batting 1st–3rd
         4.05   if batting 4th–6th
         3.75   if batting 7th or lower

P₀ = 1 − (1 − p_ab)^exp_ab

Leadoff hitters get roughly half an extra plate appearance per game over the bottom of the order, and across a season that is a large edge for a question of the form "did it happen at least once today". This is the single most under-weighted factor in casual versions of this problem: people compare hitters and forget that one of them bats ninth.

Step four: features, in logit space

Nine adjustments then modify the base rate — strikeout rate, BABIP, on-base percentage, the pitcher's K/9 and WHIP, recent form over the last 15 games, the platoon split, home/road, and batter-versus-pitcher history. Each is expressed as a standardised deviation and clamped to ±2.5, so no single feature can dominate.

They are applied in logit space, not to the probability directly:

z = A + B·logit(P₀) + Σ wₖ·fₖ
p = sigmoid(z)          capped at 0.93

This is the part worth stealing. Adding adjustments to a raw probability lets it walk out of [0,1] and treats a move from 0.50→0.55 as equivalent to 0.94→0.99, which it very much is not. In logit space the same additive nudge is automatically gentler near the extremes, which is the correct behaviour and comes for free.

Batter-vs-pitcher history gets an extra weighting of ab/(ab+15) — because 2-for-4 lifetime against a pitcher is noise, and the feature should say so.

The loop: grading, and refitting

Every board is archived with its features and raw logit. The next night, a second job pulls the box scores, marks each hitter 1 or 0 for whether they got a hit, appends the labelled rows to a history file, and computes the day's Brier score — mean squared error between predicted probability and outcome — which is a calibration check, not an accuracy one.

Then it refits, on the entire accumulated history:

warm-started, L2-regularised logistic regression
400 epochs · lr 0.05 · λ 0.02
B clamped to [0.2, 2.5] · feature weights to ±0.6
regularised toward the hand-set priors, not toward zero

The regularisation target is the interesting choice. Shrinking toward zero would mean the model forgets what baseball is whenever the data is thin. Shrinking toward hand-set priors means a few hundred graded rows can adjust the weights, but cannot invent a belief that, say, strikeout rate does not matter. The priors encode domain knowledge; the data is allowed to refine them, not overrule them. The clamps are the same instinct expressed as a hard stop.

The B coefficient is a calibration slope on the base rate itself: if the model is systematically overconfident, B drifts below 1 and compresses everything toward the middle. It is the single most informative number in the file.

What this is not

It is a box-score model. It has no batted-ball data, no pitch-level information, no weather, no bullpen modelling, no injury awareness beyond the posted lineup. When a lineup has not been published it falls back to the nine hitters with the most at-bats, which will occasionally be wrong in an obvious way.

The cap at 0.93 is an admission: no hitter is ever 97% to get a hit, and a model that says so is telling you about its own overconfidence rather than about baseball.

What it does have is the property most published forecasts lack — a scoreboard. Every prediction is recorded before the outcome is known, graded against reality, and folded back into the weights. That loop is the actual artefact. The baseball is a convenient excuse to run it in public, every day, where being wrong is visible.