Someone else's ruler

Two weeks ago I wrote up the evaluation for a retrieval system I built over a music knowledge graph: 43,567 rooms, one per artist or style, and an oracle called Pythia that answers questions by fetching rooms and writing prose from them. The headline was recall@10 of 0.89 across 115 questions in thirteen shapes: of the rooms that should answer each question, 89% arrived in the top ten.

The obvious objection, and the first one a reviewer raised before this went out, was one I did not have a good answer to.

You wrote the 115 questions. You also built the thing they grade. So 0.89 is not a score, it is a description of your own expectations.

That is fair, and it is worse than it sounds. My gold set grew by absorbing failures: when the system missed something, the something became a test case. That is a good way to stop regressions and a terrible way to learn what you do not know. A test set assembled that way converges on the shape of its author’s imagination.

The fix is a held-out set, and I had ruled one out before launch because it meant asking people to sit down and write questions for me. What I had missed is that other people already did it, published the results, and put a license on them.

1 · Three sets, and what each one is for

I took three, chosen because they break in different directions.

The third is the one I care most about, and the reason is structural. In the first two, the room I need is named in the question. “Where was Thad Jones born?” contains Thad Jones. On Reddit, nobody names the answer: they describe a feeling and thirteen strangers reply with artists. There is nothing to look up.

Grounding them against my corpus cost nothing clever. ArtistMus matched on artist names: 305 of the 500 artists have a room, giving 610 usable questions. Mintaka joined on Wikidata ids, which my graph already uses as keys, so no judgment was involved at all: 2,085 questions kept. Reddit2Deezer resolved 23,164 threads where at least three recommended artists had rooms.

2 · The predictions, written first

Before running anything I wrote down what I expected, with the thresholds that would count as bad news, and committed it before the first run. It is cheap: one paragraph, written first, removes the option of deciding afterwards what the number meant.

Three of them mattered.

• ArtistMus: hit@10 between 0.90 and 0.97. (Hit@10 is recall@10’s looser cousin: did at least one right room arrive in the top ten.) Every question names its artist, so this is a lookup and it should be easy. Below 0.85 would mean the name channel has a real gap. The system has several ways of fetching rooms, chosen per question; I call them channels, and the one that matches on names is the easy one here.
• Mintaka: the worst types will be intersection and superlative. If either falls under 0.50, the graph traversal is not being reached for questions about membership, and that is a routing bug.
• Reddit2Deezer: hit@10 between 0.55 and 0.75 on “artists like X” requests. The graph has related artist edges. They should reach some of what the crowd said.

3 · Two agreed

ArtistMus came in at 0.97, at the top of the predicted band. Contextual questions scored 0.97 and factual 0.98, a gap of 0.01 against a predicted 0.05. Nothing to report, which is what a confirmed prediction looks like.

Mintaka split the way it should. Where the question names the subject (“How old was Taylor Swift when she won her first Grammy?”), hit@10 was 0.94. Where the answer itself is a room that the question does not name, 0.67.

And one type fell through the floor I had drawn in advance: superlative, 0.43, under the 0.50 line. “Who is the youngest person in Tribe Called Quest?” names the group and wants a member, so nothing in the question points at the room that has to arrive. That is the routing bug the prediction described, called before the number existed.

The prediction also named intersection as the other weak type. Intersection scored 0.82, comfortably mid table. So the threshold was right and the reasoning was wrong: what breaks retrieval is not questions about membership, it is questions that do not mention the room you need. An intersection names both constraining entities. A superlative names none.

I would not have found that distinction by grading myself, because I would not have written enough superlatives to notice.

4 · The third one broke

Reddit2Deezer scored hit@10 of 0.05 against a predicted 0.55 to 0.75.

A miss that large is more likely a broken test than a bad system, so before believing it I counted where each request was being routed, which took a few seconds and cost nothing. 416 of the 900 requests, 46%, went to a channel that returns nothing when the system runs without an API key, which is how the offline evaluation runs. They scored zero by construction. Rescored on the requests that reached a working channel, the number was 0.09.

So the measurement was half broken. The other half was real, and it was worse than a low score, because reading what the system actually said explained it immediately. Asked this:

I’m building a playlist of unique, gritty, sleazy, and funky Americana tracks that feel older or have that vintage vibe.

Pythia answered with seven genres. Outlaw country, roots rock, southern rock, Americana, swamp rock, country soul, Texas blues. It is a genuinely good paragraph about a territory. The thread’s actual repliers had named Sean Hayes, All Them Witches, Radio Moscow, Humble Pie, Creedence Clearwater Revival, Tom Waits and seven others.

Across 25 requests of that kind, it cited 170 style rooms and zero artist rooms. Asked to recommend artists, it was describing genres.

The corpus was not missing the answer. Style rooms already carry ranked lists of their artists, and three of the thirteen artists the crowd named were sitting inside the seven rooms it had just cited: Creedence Clearwater Revival in the top tier of swamp rock, ZZ Top in the second tier of southern rock, Tom Waits in the third tier of Americana. The channel stopped one hop short: it cited the rooms instead of reading them.

That was a small fix, reading data already loaded, and it took hit@10 from 0.133 to 0.240 on real requests with the rest of the evaluation unchanged. For scale, answering every request with the ten most followed artists in the corpus and ignoring the question entirely scores 0.013.

I would not have found that by grading myself either. My own gold set for situation questions expects genres: ask it for music for when you are feeling sad and the right answer, according to me, is sadcore, slowcore and emo. The system scored 0.96 against that. It was answering the question I had written, correctly, and the question was the wrong shape. Strangers asking the same thing want artists, and had wanted artists 186,380 times before I started.

5 · The number that is not about the oracle at all

Three of these results are about coverage, and coverage is a different system’s problem: the scraper and the graph, not the retrieval.

• 305 of 500 artists on a deliberately globally balanced list have a room. 61%.
• 76% of the artists Reddit actually recommends have a room, though only 27% of the unique names do, because the long tail is what is missing.
• On ArtistMus the fact needed to answer was present in the room text 32% of the time.

Michala Petri, Catupecu Machu, Connan Mockasin, Xenia Ghali and 191 others are not in the graph. No amount of retrieval tuning fixes that. What the three sets handed me is a shopping list, written by strangers, for the next six months of scraping.

6 · What it does when it does not know

The multiple choice set turned out to measure something I had not planned for.

Given four options and told to answer with a letter, Pythia declined 59% of the time, in terms that name a reason:

I have no rooms about Thad Jones or his move to Copenhagen in 1979. My rooms contain nothing about him, his decision-making, his work with any big band, or his career trajectory. I cannot answer this question.

Of the 41% it did answer, 81.6% were right. Then I ran the same questions through the same model with no retrieval at all, as a control: it answered more often, 58%, and was right 60.9% of the time it did. The graph makes the same model more careful and more accurate, and on a benchmark that rewards guessing, being careful costs you the headline.

That raised an obvious worry. If it is right 81.6% of the time, how much of that is the model simply knowing, rather than the rooms telling it? Checking took a control of its own, and the short answer is 1.7%: asked the same questions without the four options, the way a person asks, it stated a fact the rooms did not hold twice in a hundred and twenty. The test, and the control that makes it mean something, are in The control did the work.

That is the property I would want if this were my data: it would rather say nothing than make something up. It is not a guarantee, 1.7% is not zero, and I would not print the word guaranteed. But it is a measurement rather than a promise, and the test that produced it takes an afternoon.

7 · Where it still is not good

Mintaka’s answers, as opposed to its retrieval, are weak. The rooms hold the answer about three quarters of the time and the system says it under a third of the time. Comparative questions were the worst of all, and reading them shows exactly why:

Q. Who is older, Nicki Minaj or Iggy Azalea? A. My rooms contain Nicki Minaj’s birth date [1], but I have no room for Iggy Azalea. I cannot answer this question.

I wrote a paragraph here explaining that a comparison needs both sides, and that if each artist independently carries a birth date seven times out of ten, the comparison can only work about half the time. It reads well. It is also wrong, and I only found that out on a reread of the draft, when I stopped at this quote and thought: we absolutely have a room for Iggy Azalea.

We do. It is Q2748803. It says she was born on June 7, 1990.

My second explanation was that the system had the answer and refused anyway. I had checked, after all: run that question through the search and Iggy Azalea’s room comes back at rank one, its first line reading “Amethyst Amelia Kelly (born 7 June 1990), known professionally as Iggy Azalea.” Nicki Minaj’s room comes back at rank two with her date. Both dates in the prompt, and the answer is 1982 against 1990.

That explanation was also wrong, for a reason worth more than either of the things it replaced. I had run the search the way the evaluation runs it, and the evaluation does not run it the way the product does.

Without an API key, the system routes with fixed rules and this question goes to plain search, which returns both artists. With a key, an extra step routes it instead, and that step sends comparisons to the part of the system that walks the graph. That part resolves the first artist named, follows her stylistic edges, and never looks up the second one at all. So in production, this question fetches:

Nicki Minaj, hip-hop, contemporary R&B, pop music, T-Pain

Iggy Azalea is genuinely not there. Across twenty comparison questions, the version I measured fetches everything the question names 95% of the time, and the version that actually runs manages 35%.

So the refusal was truthful about what the system had been handed, and false about what it holds. Those are different sentences, and only one of them was said. “I have no room for Iggy Azalea” is a claim about the corpus and it is wrong; “Iggy Azalea is not among the rooms I have here” would have been right, and would have sent the reader to look rather than to conclude the graph was thin. That distinction is the entire difference between a system you can check and one you have to trust, and I got it wrong in the one place it shows.

The uncomfortable part is the other half. The number I had recorded for this shape, 0.99, was measured on the path production does not take. The harness even prints a warning that live routing differs, which I had read for weeks as a note about coverage. It is not. It is the measurement flattering the system, one more instance of a failure I had already counted six times in this project, and the first after I thought I had learned the lesson.

I checked how far that spreads before writing this, because the honest version of a scare is a measurement. Routed the way production routes, ArtistMus loses 0.017 and Mintaka 0.067 across sixty sampled questions each. So the top-line numbers survive, slightly reduced, and comparisons are the one shape that falls off a cliff. Everything in section 6, the refusal rates and the invention rate, was measured live to begin with and is unaffected.

What it cost to fix

Three lines. The bug was in what one step handed the next.

The step that routes the question was doing its job perfectly. Given “Who is older, Nicki Minaj or Iggy Azalea?”, it reports back that this is a lookup, that the first entity is Nicki Minaj, and that the second entity is Iggy Azalea. It had found both all along. The next step, the one that turns that report into a fetch, read the first slot and ignored the second.

So: a step that can only fetch one entity, handed two, now declines and lets the question fall through to plain search, which fetches both. That guard already existed a few lines above for a different two-entity case, with a comment explaining why. It simply had not been applied to the channels that drop the extra name silently rather than loudly.

Comparison questions now retrieve every entity they name 95% of the time, up from 35%, which is exactly what the offline path was getting all along. Overall recall across the whole test set moved 0.8239 to 0.8242, which is to say not at all. And the question that started this now answers:

Q. Who is older, Nicki Minaj or Iggy Azalea? A. Nicki Minaj is older. She was born December 8, 1982 [2], while Iggy Azalea was born June 7, 1990 [1]. That makes Nicki Minaj nearly eight years her senior.

Two more things went wrong on the way. My first attempt at this fix detected the second entity with a pattern over capitalized words, and it resolved the “Who” at the start of every question to the band The Who. It was also unnecessary, because the second name was already sitting in the routing report I had not read closely enough. And one shape appeared to get worse by 0.185, which looked like a real cost until I ran the same measurement twice with the fix turned off and got 0.393 and 0.220. The routing step calls a language model, so it is not deterministic, and a three question shape cannot detect anything smaller than the noise. That is the fifth number in this piece that needed a control before it meant anything.

The wording changed too. A refusal now has to be about the rooms in front of it rather than about the whole graph, because those are different claims and only one of them was ever true.

Superlatives are worse still, and for a cleaner reason:

Q. Who has won the most Grammy Awards of all time? A. My rooms do not contain information about who has won the most Grammy Awards of all time. They mention specific achievements, Beyoncé winning Album of the Year at the 67th Annual Grammy Awards [3], Adele winning six awards in a single night [5], but they do not identify an overall record holder across Grammy history.

Nothing is wrong with the corpus there. The question wants an operation over all of it, and that operation does not exist yet.

What I would take from this

If you have a large and confusing pile of data and you want a language model to open it up without inventing things, this is the shape of the work, and most of it is not model work.

1. Grade yourself on questions you did not write. Everything above that was worth knowing came from the three sets, and the two most useful findings came from the one that scored worst. Published benchmarks with a license exist for most domains, and grounding them against a corpus took a name match and a join on ids.
2. Write the predictions down first, including the thresholds for bad news. The superlative bug was called before the number existed, which is the difference between a finding and a rationalization. It also removes the temptation to quietly reframe a result you did not like.
3. Measure how often it refuses, and check what it says when it does. A system that answers everything is not better than one that answers 41% of the time and is right 82% of those; it is differently wrong, and only one of them tells you when to go and check. But make it say “that is not in what I fetched” rather than “I do not have that”. The first is checkable and usually true. The second is a claim about your whole corpus, and mine made it wrongly about a room it owns.
4. Read the output, and get somebody else to read it too. The best findings on this page came from reading answers, not from reading numbers. A benchmark tells you something is wrong; it rarely tells you what. The most useful sentence in this whole exercise was six words long and I said it to myself, rereading one quoted answer: we absolutely have a room for Iggy Azalea. Two of my own explanations died that afternoon, and a second reader would have killed them sooner.

Every number here was misleading until a control corrected it, and that pattern turned out to be its own note: The control did the work.

Pythia is live at ask.orpheus.rocks. Ask it something it does not know, and see what it says.

Appendix: the numbers

Measured 2026-08-26 by calling the shipped code. Retrieval runs cost nothing; the answer runs cost $2.19 in total.

Read the first four rows with section 7 in hand. Every “right room in the top ten” number here is measured with the routing the harness uses, which is not the routing the live product uses. I then measured the gap on a sample of sixty questions per set, and it is smaller than section 7 might suggest: ArtistMus loses 0.017 and Mintaka loses 0.067 when routed the way production routes. Comparison questions were an outlier rather than the pattern, because they were the shape sent to a step that resolved only one entity. That one is fixed, and the rows below have not been remeasured since.

That is a small correction and a real one, and it points the same way as everything else here: the direction of the error was flattering.

Reddit2Deezer’s 0.24 is after the fix described in section 4, and under a ranking that was corrected the same day; the corrected ranking projects 0.227 and has not been rerun. Mintaka’s answer accuracy is reported in the linked working notes rather than here, because its scoring metric is inflated in a way I am still bounding: between 0.215 and 0.310 depending on how strictly you count an answer that names the right artist while declining to compare them.

Next in the series: The control did the work →

Siegfried Martens, 26 August 2026. Part of the Daedalus notes: what was built, what it measured, and what the measurement could not see. Questions and corrections to daedalus@s-martens.com.