A benchmark told me my recommendation engine scored 0.05. Counting where the questions went, which cost nothing, showed that 46% of them were being scored against a component that returns nothing when run offline.
A metric said precision fell after a fix. It had not. The metric was only defined on 14 of 75 cases before and 70 of 75 after, so the two numbers were averages over different populations.
The most useful control of the day was rereading one answer and not believing it: “we absolutely have a room for that.”
In one day I ran three published benchmarks against a retrieval system I built: a music knowledge graph of 43,567 rooms (one page per artist or genre) and Pythia, an oracle that answers questions from them. I wrote predictions before each run so I could not reinterpret the results afterwards. That part worked, and I have written it up elsewhere.
This is about something I did not expect. Every headline number those benchmarks produced was misleading, and each one was corrected by a control that took minutes and usually cost nothing. Not one of the four conclusions I would have drawn from the raw scores was right.
The pattern is worth more than the project it came from, so here are the four, in the order they happened.
1 · The score that was measuring a switched-off component
The recommendation benchmark came back at 0.05 (the share of requests where one of the crowd’s artists made the top ten), against a prediction of 0.55 to 0.75. That is a twelvefold miss, and my first instinct was that the system was simply terrible at recommending.
Before believing a number that far out, I counted where the questions were going. The system has several ways of answering, chosen per question, and one of them expands a mood into genres using a language model. Running the evaluation offline, without an API key, that component is replaced by a stub with a small cache of prepared answers. It has no entry for arbitrary text off Reddit, so it returns nothing.
416 of the 900 requests, 46%, were being routed to it. They scored zero because nothing ran.
Rescored on the requests that reached a working component, the number was 0.09. So the measurement was about half broken, and the surviving half was still bad enough to be worth chasing, which is where the real defect turned up. But “terrible at recommending” and “half your test was unplugged” call for completely different afternoons.
The control was counting a column.
2 · The regression that never happened
Then I fixed the defect, re-ran, and one number went the wrong way. Precision fell from 0.079 to 0.039. Halved. I very nearly reported it as a cost of the fix.
Precision here is the share of the system’s suggestions that the crowd also suggested. It is only defined when the system suggests at least one artist, and before the fix it very rarely did: the metric was computed on 14 of 75 requests. After the fix, on 70 of 75.
So the two numbers were averages over different populations, and the earlier one was an average over the cases where the system had already been confident enough to name somebody. Scored on the same 14 requests both times, precision went 0.079 to 0.071. Nothing had regressed.
This is the dullest of the four and the one I would most want remembered. A metric that is only defined some of the time will move when the frequency of that condition changes, and it can move in the direction that makes your work look worse or better without anything having happened. The control is to check the denominator before believing the ratio.
3 · The leak that was a property of the exam
The third one nearly reached my public copy.
The lookup benchmark is multiple choice. I split its results by whether the fact being asked about was in my corpus at all, expecting the ones it did not hold to score near the 25% you get by guessing between four options.
They scored 81.5%. On the questions whose answer my data does not contain, the system answered 119 times and picked the right option four times in five. Which means those answers were not coming from my data. They were coming from the language model’s own memory, and roughly half of all its correct answers were arriving that way.
That is a bad finding for a system whose entire pitch is that its answers are grounded in documents you can click through and check. I drafted a paragraph saying grounding was a tendency rather than a guarantee.
The control was to ask the same questions without the four options, the way a person actually asks. If the model is reciting from memory it will keep doing so.
It did not. On the same facts, asked openly, it stated one 1.7% of the time. Two cases in a hundred and twenty. And to prove the test could detect an answer at all, I ran the same check on facts the corpus does hold: 45%.
4 · The metric that scored a refusal as correct
The fourth is the worst, because the number was flattering and I published it internally before catching it.
On the counting-and-comparing benchmark, the strongest category was comparisons: 0.800, well above what I had predicted. I wrote a paragraph explaining why, something about the system being good at facts it stores directly and weak at facts it has to compute.
Then I read one of the answers it had marked correct.
Q. Who is younger, Billie Eilish or Olivia Rodrigo? (answer: Olivia Rodrigo) A. I don’t have information about Olivia Rodrigo’s birth date in my rooms. I know [1] that Billie Eilish was born December 18, 2001, but I cannot answer your question without data on Rodrigo.
The scoring works by checking whether the correct answer appears in the response. “Olivia Rodrigo” appears in it. It appears inside the sentence explaining that the system does not have her.
For a comparison, the right answer is one of the two names the question already contains. A system that simply echoes the question scores. Once I excluded the 25 questions with that property, comparisons fell from 0.800 to 0.143, from the best category to nearly the worst, and my explanation of why it was strong evaporated along with the strength.
That episode cost me two wrong explanations before I got to a right one, and I would not have started looking at all if I had been reading the scores instead of the answers.
What the controls actually were
Nothing here required a better benchmark or a bigger budget. The complete list:
| control | what it cost | what it settled |
|---|---|---|
| count which component each question went to | nothing | 46% of a test was being scored against a stub |
| rescore both runs on the same subset | nothing | the “regression” was a denominator |
| ask the same questions without the options | about $0.30 | the leak was the exam, not the product |
| exclude questions containing their own answer | nothing | the best category was the second worst |
| run the same model with no retrieval at all | about $0.03 | the graph makes it more careful and more accurate |
| answer every request with the ten most famous artists, ignoring the question | nothing | a floor of 0.013, so the real gain was real |
Four of the six cost nothing. The entire day, benchmarks and controls together, came to $2.19. The expensive part was never the measuring.
Two things a score could not have told me
Both came from reading answers, and the first is told in full in the companion piece.
My own test set encoded the wrong shape of answer. Asked what to play when you are feeling sad, my hand-written gold answer is sadcore, slowcore, emo. Genres. Real people asking that question on Reddit want artists. The test did not merely contain its own answers; it agreed with the system about what a good answer even looks like, and no amount of scoring against it could surface that.
And after a second fix, the benchmark preferred the broken version. This one changed how the system ranks the artists inside a genre once it has decided to name them. The better ranking gives Michael Jackson, David Bowie and Depeche Mode for an eighties dance party; the worse one gives Taylor Swift. The worse one also scores 0.240 against the crowd where the better one scores 0.227, because a crowd naming artists names famous ones. The gap is one request in seventy-five, well inside noise, and it would still have pointed the wrong way if I had been optimizing rather than reading.
The best control was a person
The first draft of the companion piece quoted this, as an example of the system honestly admitting a gap:
A. My rooms contain Nicki Minaj’s birth date [1], but I have no room for Iggy Azalea. I cannot answer this question.
Rereading it, I stopped there. We absolutely have a room for Iggy Azalea.
We did. It ranks first for that question. The cause was a routing step that had correctly identified both artists and then read only the first of them, and the fix was three lines; comparison questions went from retrieving what they need 35% of the time to 95%. The whole story is in the companion piece.
Nothing in that table found it. What found it was reading one answer and refusing to believe it. The controls caught four things no reader would have, and the reread caught the one thing no control did. The output is evidence, and most people never look at it, because a table of scores is so much easier to read than four hundred paragraphs of prose.
What I would take from this
If you are having a system like this built, or building one:
- Treat a benchmark result as a candidate, not a verdict. It is good at telling you that something is wrong in a particular area. It is close to useless at telling you what, and the obvious interpretation was wrong four times out of four here.
- Spend on controls before you spend on another benchmark. A second benchmark gives you another candidate. A control resolves the one you already have, and the ones above cost either nothing or the price of a coffee.
- Check the denominator before you believe a ratio, especially on any metric that is only defined some of the time.
- Validate the instrument on a case where you know the answer. The 1.7% means nothing without the 45%.
- Read the output, and get somebody else to read it too. Nothing else on this list would have caught the routing bug.
The uncomfortable version of all of this: I had written a piece about evaluation-first development, about the six separate times my own measurement had turned out to be wrong, and about the fact that every one of those errors flattered the system. I then made four more of the same mistake in a single day, and this time the errors ran both ways: two made the system look worse than it was, two made it look better. The only constant was that the first reading was wrong, and one of the four survived until a reread caught it. The lesson does not appear to be learnable in the abstract. What is learnable is the habit of distrusting your own headline number for the twenty minutes it takes to check it.
Pythia is live at ask.orpheus.rocks. Ask it something it does not know.
Siegfried Martens, 26 August 2026. Part of the Daedalus notes: what was built, what it measured, and what the measurement could not see. Questions and corrections to daedalus@s-martens.com.