One month of calibration testing, and the metric that passed by construction
A 30-day pilot of a daily calibration practice produced 22 sessions, a flat Brier score, one criterion that was true by definition, and eight lessons that outlived the tool I retired.
For about three weeks I answered ten true/false forecasting questions a day, from a bank of 250, and logged my confidence in each answer. The point was not the score. The point was to find out whether a daily calibration practice is a real instrument or a ritual that feels like one.
The pilot ended on 1 November. I retired the tool. These are the eight things I learned, and the reason I would run the pilot again even though I would not run the tool again.
Every number below is from the pilot’s own records. Where a number is thin enough to doubt, I say so.
1. A criterion that can only be trivially true is decoration
The pilot’s success criterion was that it would complete with a drift flag inactive across a rolling window of 20 sessions.
The flag’s own implementation only computed when the prior comparison window held at least 10 completed sessions. That is, from session 30 onward. The pilot produced 22. So the flag never computed, returned a hardcoded false, and the criterion passed.
The uncomfortable part is what it would have said. The counterfactual comparison for that window came out at 0.1631 against a flag threshold of 0.05. Had the pilot run long enough to evaluate its own criterion, the criterion would have failed, and failed by a factor of three.
2. A prevention gate is not a detection instrument
There was a watchlist of 63 regular expressions applied when the question bank loaded. It functioned as a deny-gate: if a served question matched, something would have blocked.
The per-item field the drift summary was supposed to group over (watchlist_topic) was null on all 250 items. The observable population for the report was empty by construction. The summary would have said “zero violations” forever, and that would have been both true and meaningless.
The test I now apply: for any drift or trend criterion, name the population the report groups over. If that population is empty by design, the report is a zero-row table and the criterion measures nothing.
3. A trend read off the first few observations is usually regression to the mean
The day-5 checkpoint reported a monotonic fall from 0.442 to 0.198 and called it a roughly 50 percent improvement.
Over all 22 sessions the slope was not significant. Removing a three-item smoke session and one degenerate session where every answer was “true” flattened it to p ≈ 0.44. Restricted to sessions 6 through 22, the slope had the wrong sign.
Both outliers sat inside the first five sessions, which is exactly where a first-n read looks best. This is the most seductive failure on the list, because the early improvement is real in the data and the honest answer is that it will not hold.
4. A scalar scoreboard hides the one miscalibration that matters
Mean Brier sat around 0.23 for the whole pilot. Flat, uninformative, and the kind of number that ends a conversation.
The binned shape said something specific. Fifty-five percent of items sat in the 50 to 60 confidence band at roughly 50 percent accuracy, which is a permanent ~0.25 Brier tax per item. The concentrated loss was in the 85 to 100 band, where predictions of 87.5 to 97.5 resolved at 50 to 89 percent.
The aggregate said nothing changed. The bins said one band is costing you. If the scalar is the headline, you will miss the actionable part every time.
5. Decompose before you file it as “no signal”
The headline slope was flat, but the flatness sat on two offsetting movements. Decomposing Brier into reliability, resolution, and uncertainty: reliability fell from 0.0463 to 0.0137 between pilot halves, with P(improved) ≈ 0.932, while uncertainty stayed flat. That is a real calibration gain, not an artifact of hedging harder.
Resolution fell. So I became better calibrated without becoming better at discriminating. The decomposition said both what changed and what did not. The headline said neither.
6. A statistic is not a liveness test until you have its null
This is the lesson that killed the tool.
The pilot’s underlying worry was that unattended answers might not be human. Three intuitive floors were proposed. All three failed their own nulls.
| Floor | What the null actually did |
|---|---|
| Resolution floor | A random answerer’s null median (0.0137 at n=150) exceeded my observed value (0.0163 across the pilot, 0.0217 in the second half). The floor has no discriminating power at all. |
| 55 percent accuracy floor | Rejects only about 15 percent of random answerers. An always-“true” bot scores 57.3 percent outright, because the bank was 57/43 T/F. |
| “No all-same session” rule | Null rate 0.38 percent per session, so 7.3 percent of honest 20-session runs trip it. My real run tripped it exactly once, which is precisely what the null predicted. |
What actually separates a human from a deterministic bot here: provenance (answers traceable to an operator-typed turn) and the all-same-session count (a deterministic bot produces roughly 20; a genuine answerer produces about 0.07).
7. Capacity has a date on it, and you can compute it at design time
250 items at 10 per session under a no-repeat rule is exhaustion at session 26. The phase-1 review predicted that number exactly. It was still unaddressed when the pilot ended, with the extension tool commissioned against a deadline and never built. The bank was the original phase-0 artifact, byte-identical, at 89 percent consumed.
The general form: divide the horizon by the cadence before shipping. If the horizon needs 30 sessions, the bank needs at least twice that. And if the extension path is described as “a follow-up,” it is a scheduled outage, not a follow-up. Build it in scope or drop the horizon claim.
8. “Baked into daily practice” is a claim about a habit, not a tool
Participation ran near-daily for 22 days and then stopped dead. A reconnect on day 32 produced no session. The tool was flawless throughout, which is the entire point: the tool’s health told me nothing about the habit’s health.
Two consequences. Measure adoption as a count of unattended invocations with provenance, never infer it from the tool existing. And size the observation window so an abrupt stop is visible: a 21-day window would have passed the very failure it was built to catch, and even a 28-day window only catches it if the bar includes a clause requiring the final days to still be populated.
Before fixing a window length, re-run the old data against the new criterion and check whether the bar would have caught the failure you are trying to prevent.
What I think this adds up to
The pilot cost a month and produced two things: a set of falsification habits that now sit in front of every measurement I build, and a demonstration that most of my intuitions about what “real” data looks like were wrong in the specific way that flatters the experimenter.
The tool is gone. Every lesson here is aimed at the general case: any time you point an instrument at yourself, the instrument’s own construction is the most likely source of the answer you get.
What would make this wrong
The sample is 22 sessions from one person, so any claim about my calibration is weakly supported and I am not making it. The 0.1631 counterfactual depends on a comparison window that never ran, so treat it as a warning sign rather than a result. And the nulls in lesson 6 were computed at n=150 on a bank that is 57/43 T/F: a more balanced bank would move the always-true bot’s score and weaken that particular argument, though not the general one about provenance.