The evaluation was built on day two, before any retrieval channel existed: recall@10 0.40. Two days after launch, measured by calling the shipped code: 0.89, hit@10 1.00, MRR 0.94, across 115 questions in thirteen shapes.
Six times the harness measured something other than what ships. Every one of the errors flattered the system.
What to ask anyone building you a retrieval system: the baseline, the blind spots, and whether the eval calls the code you ship or a copy of it.
Most AI demos show you a good answer. That is the easiest artifact in the world to produce: ask enough questions, keep the one that worked, screenshot it. This piece is the ruler instead: the number that says how often the right document showed up at all.
The system is a retrieval-augmented one, the shape most companies mean when they say “AI over our own data”: a language model that is not allowed to answer from memory, only from documents fetched out of a corpus at the moment you ask. That design moves the hard problem. The model’s prose is rarely the weak part; the fetching is. If the wrong documents arrive, a fluent answer is built on them and sounds exactly as confident as a right one. So the number that matters is not the prose.
Pythia is an oracle over Orpheus, a knowledge graph of musical culture built from Wikipedia, Wikidata, MusicBrainz and Spotify. You ask it a question in plain language and it answers in prose, with every claim linked back to the room it came from so you can check it. It went from an empty directory to answering questions on the public internet in three weeks. What follows is what the measurement said, including the six times it was wrong.
Two things happened before any of that was possible, and they say what a knowledge graph of forty thousand musicians is like inside. Ranking the candidate musicians by how widely they are read put Sydney Sweeney, Pope Francis, Rob Reiner and Tom Cruise near the top, and every one of them qualifies on a technicality: Scarlett Johansson has two albums, Keanu Reeves spent a decade in Dogstar, and Francis released a prog-rock record in 2015. And because Wikipedia lists band members by name, and the resolver bound each name to the most famous person who had it, Richard Wagner was recorded as a member of The Impalas and Robert Johnson of KC and the Sunshine Band. The tool that found that class looked for band members who had died before their band existed. Wagner died in 1883. The Impalas formed in 1958.
1. The eval existed before the system did
Day two, before a single retrieval channel was written, I built a gold set: real questions with the rooms that should answer them, hand-grounded to real entities. Then I ran the only channel (a way of fetching candidate rooms) that existed, plain semantic search over the embedded corpus, and wrote down what it scored.
Recall@10: 0.40.
Recall@10 is the share of the rooms that should answer a question which actually show up in the top ten results. 1.00 means all of them did; 0.40 means most of them did not.
That number is the most useful one in the project, because it is bad. Every claim I make below is legible only because that baseline existed first. You cannot tell whether you improved something you never measured, and more to the point, you cannot discover that your ruler is too short if you never had one.
2. Four zeros, four channels
The gold set covered eight question shapes, and the aggregate 0.40 hid a structure that turned out to be the entire plan. Semantic search did fine on questions about meaning. It scored zero on the ones that are not:
| kind of question | example | semantic search only |
|---|---|---|
| How two things relate | “How does ambient relate to minimalism?” | 0.87 |
| Where and when | “Where and when did grunge emerge?” | 0.50 |
| Who or what belongs here | “What are the essential krautrock bands?” | 0.48 |
| Counting and ranking | “What styles have the most artists?” | 0.03 |
| Music for a situation | “Music good for getting focus work done” | 0.05 |
| Puzzles and constraints | “Which artists have exactly three A’s in their name?” | 0.00 |
| Facts about the names | “Bands with ethereal, ghostlike names” | 0.00 |
Embeddings cannot answer those four in principle. “What styles have the most artists” is arithmetic. “The opposite of peaceful” requires negation, and a vector index cannot negate: it returned ambient music. “Bands with long, complicated names” is a fact about the strings, and the index returned heavy metal and The Long Ryders.
The shapes were the build order. A bounded compute layer over a precomputed metadata table took counting and puzzles from 0.03 and 0.00 to 1.00. Inverting the flow for mood questions, letting the model propose genres and then forcing every proposal to resolve against the real corpus, took the situation questions from 0.05 to 1.00, and the negation question started answering noise, grindcore, industrial, powerviolence, harsh noise wall. A lexical channel took the name questions from 0.00 to 1.00, ranking by length times prominence, because pure length surfaces obscure orchestras and pure prominence surfaces the Red Hot Chili Peppers. For bands with long, complicated names it produced The World Is a Beautiful Place & I Am No Longer Afraid to Die, which is a better answer than the one I had written in the answer key. Deterministic graph traversal over the room edges took the relational questions.
The failures on the other side were not subtle either. Asked for the earliest forms of rock and roll, the system answered Philoxenus of Cythera, a Greek dithyrambic poet of the fifth century BC, along with the Minnesänger. It was not malfunctioning. The poets are in the corpus as influence nodes, so asked for the oldest things it knew about, it sorted everything it held by date and answered the question it was asked, which was not the one I meant. There is now a guard in the codebase that the project notes refer to as the medieval-poets guard.
Four new channels in 48 hours took combined recall from 0.40 to 0.77. Once the measurement says which four shapes are broken and why, building is the easy part.
What the machinery ended up being
There are eight ways the system can find an answer, and the badge a visitor sees names not the channel but one of four kinds of evidence (the channel itself sits behind a detail toggle), in these words, which are the words it says:
- “followed the edges between rooms” for anything about how things connect, which covers the graph, the city scenes, and the clusters
- “counted across every room” for anything arithmetic, which covers counting and grouping
- “listened for meaning across the rooms” for the ordinary semantic search everybody pictures when they hear AI, plus the mood questions that get turned inside out first
- “weighed the names themselves” for questions about the text of a name rather than the music behind it
It can also decline.
3. Two regimes, and the sentence that separates them
The first regime was capability. Every gain came from something that did not exist before. Build a channel, watch a shape go from zero to one. Large effects, easy to see.
The second regime was reach. No new channels at all. Every gain came from discovering that something already built was not reaching the questions it was built for:
- The router’s phrase list was narrower than its own ops. The graph already held the answers to “what are the essential krautrock bands”; the router just never sent it there. Rewording the question to a phrase already in the list took it from 0.20 to 0.80, which proved the retrieval was fine and the door was locked.
- Descendant edges were never ranked. “What did hip hop spawn” returned the first ten of 131 edges in storage order, which is an arbitrary ten, which is why it answered Alaskan hip hop and Bangladeshi hip hop.
- The section-grounding pass was discarding the answering section on every question of one shape.
Nothing was missing; things were unreachable. That has a consequence.
4. The plateau was in the measurement
For eight days the number sat at 0.80 and would not move.
For five of those days I was not working on the oracle at all. I was rebuilding the corpus underneath it: roughly 36,300 artists to over 40,000, and 39,080 rooms to 43,567. There are no retrieval commits at all in that stretch. So the number was flat because nobody was moving it, and the fact worth keeping is the other one: it held at 0.80 while the ground under it grew by 11.5%. It is the only stretch where the system was asked to survive a moving substrate instead of improve on a fixed one.
The real plateau was the three days after, when work resumed and the number still did not move. The natural reading is that the work was worthless. The real reading was worse: five different ways of choosing what text to index, differing by 72,559 chunks of content between them, had all scored identically, to every decimal the harness printed.
A measurement built on sixteen sections across seven questions moves in steps of about six percent, and anything smaller is invisible. In the first regime that is fine, because the effects are enormous. In the second regime every remaining effect is smaller than the ruler’s resolution, and a flat line means nothing at all.
Widening that slice to 27 questions and 47 sections immediately adjudicated two changes it had previously scored as noise, and found four live bugs on its own, including two router mis-routes that were returning zero correct rooms to real questions.
A metric that stops moving is a claim about the metric, not only about the system.
5. Six times the ruler was wrong
Six times, the test was not testing the thing that ships. A test can be wrong in two ways. It can measure the right system and get the number wrong, which is the failure everyone imagines. Or it can measure something that is not the system at all, and report a perfectly precise number about a thing nobody uses. All six below are the second kind, and the second kind is invisible: the number looks fine, so nothing prompts you to check.
-
A metric measuring an algorithm that production does not run. The eval scored a global top-k across the whole chunk index. Production retrieves rooms first, then looks inside each one. Reported: 11 of 16 sections reaching the composer. Actual: 7. A quarter of the metric was an artifact.
-
A harness that reimplemented the thing it was testing. It needed a selection rule, so it copied one, and got two details wrong. It reported 253 chunks where production sees 120.
-
An eval scoring an index nobody serves. It measured all 246,116 chunks while production loaded 173,557. Its docstring said it matched production. It did not.
-
A bug report about behavior the system does not have, because a component was observed below the router that decides whether it ever runs.
-
The primary harness could not see production at all. This one turned up on the day the system went live, and it turned up by accident: a ranking fix went in, the gold set was re-run, and the numbers came back identical. The reason was that the harness reimplemented the entire router instead of calling it, so the function that had just changed was never executed. The headline figure everyone quotes had been measured against a copy of the system.
-
A whole shape the harness could not report on. The per-shape block iterated a hardcoded list of twelve shapes that omitted
deep_content: 27 questions, the largest shape in the set, 23% of the gold questions. It counted toward the overall score and was missing from the per-shape table, so its absence read as not yet measured and no amount of re-running would ever have filled it. Like the fifth, this one surfaced by running the thing rather than by reading it. And when the row finally appeared it read 1.00, which is a ceiling (see the appendix): those questions are answered by a section, and room-level recall structurally cannot see that failure mode.
There is one more worth adding, outside the six, because it is the same joke told by a different instrument. A geocoding fix described in the next section was a rule: reject a location that shares no word with the source text the claim came from. It shipped, and it passed the very case it had been written for, because “Vancouver, British Columbia” and “British West Africa” share the word British. The word that caused the bug also cleared the guard against it. Colonial adjectives are now on a stoplist, and the class survives one level in: Cajun music still resolves to Spain, because its source text says New Spain, and unlike British, Spain is a real place name that cannot be banned.
Fixing the fifth exposed a seventh inside it. The copy’s top-up logic appended up to ten extra rows, so recall@10 had been scored over as many as twenty candidates. Correcting it moved two shapes down and the headline from 0.90 to 0.89.
The rule I now hold, written after instance four and violated by instance five: an eval calls production code, never a copy of it, and a claim about what the system returns is made at the entry point, not at a component underneath it. Where an eval cannot do that, it says so in its own docstring.
Note what all of them, the six above, the seventh hiding inside the fifth, and the guard, have in common. Every one made the system look better than it was, or made a real change look like no change. Measurement error is not randomly signed. It flatters.
6. The number, and what it still cannot see
Two days after launch, measured by calling the shipped entry point:
Recall@10 0.89. Hit@10 1.00. MRR 0.94. Across 115 questions in thirteen shapes.
In words: of the rooms that should answer each question, 89% arrive in the top ten; every question gets at least one right room in its top ten; and the first right room sits at or near the top almost every time. The per-shape scores are in the appendix.
Three caveats:
- It is not comparable to the earlier ones. The gold set grew from 59 questions to 115, and it grew by absorbing failures. On one representative day the headline read 0.87 to 0.91, but twenty of those questions were new and scored 0.90 before any fix landed. The same-set gain that day was 0.820 to 0.849. A real gain, about a third of what the headline implied.
- It does not cover the metered rescue router. That path costs money per call, so the offline eval runs without it, which means roughly a third of the questions route differently in production than in the harness. It is documented. It is not fixed.
- There is a class of defect it scores 1.00 and cannot see. A geocoding bug once had the oracle report highlife, a Ghanaian genre, as originating in Vancouver, because the geocoder matched “British West Africa” to “British Columbia”. The correct room ranked first. Recall was perfect. The answer was wrong. No retrieval metric will ever catch that, which is why the gold set is not the only instrument.
Two more of the same kind.
A live screenshot showed Punchmade Dev, a scam-rap figure from Lexington, Kentucky, carrying an Opera tag, fifty opera singers as his related artists, and an origin of Heard Island and McDonald Islands, an uninhabited Antarctic territory. Someone had vandalized his MusicBrainz entry. The pipeline believed all of it, and he arrived in the Opera room’s top artists next to Callas and Pavarotti. Retrieval was perfect: it was asked for opera and it returned the Opera room.
To stop people using a music oracle as a free general-purpose chatbot, it refuses questions containing words like code, essay and translate. Nobody checked those words against 43,567 rooms named after ordinary English things. So it refused “who are the members of Code Orange?”, and, because her surname contains essay, “tell me about Natalie Dessay.” A stranger asking about a hardcore band was told the oracle only answers questions about music. There is not one refusal in the gold set, so nothing was measuring that in either direction.
Most of the real defects were found by asking the system questions and opening the data on any answer that felt off. The eval’s job is to keep the fixes fixed, so the next change does not undo three of them.
What I would take from this
If you are evaluating someone to build this kind of system, the demo tells you almost nothing. Ask for the eval, and ask three things about it:
- What did it score before you started? If there is no baseline, there is no evidence of improvement, only assertion.
- What can it not see? Every measurement has a hole; a good one says where. An evaluation with no stated blind spots has not been examined.
- Does it call the code you ship, or a copy of it? I would not have thought to ask this a month ago. It is on the list because a harness in this project failed it six times in one month, and every one of those failures made the system look better than it was.
One thing the graph did that nobody asked for.
Asked how the Beatles connect to the Rolling Stones, the graph returned a chain nobody put there: The Beatles → John Lennon → The Dirty Mac → Keith Richards → The Rolling Stones. The Dirty Mac existed for one night in 1968, assembled for the Rock and Roll Circus, and it is a real edge between two rooms because Lennon and Richards really were in a band together, once. Nobody wrote that path down. It was there in the data, and the machinery walked it.
Pythia is live at ask.orpheus.rocks. Ask it something, and click through on anything it tells you.
Appendix: recall@10 per shape
Measured 2026-08-14 by calling the shipped entry point, one run without the metered rescue router. Hit@10 is 1.00 on every shape: every question retrieves at least one correct room in its top ten, and all the variation below is in how many of the expected rooms arrive.
Each row is a kind of question, and every example below is taken from the test set. n is how many of that kind the set holds.
| kind of question | example | n | recall@10 |
|---|---|---|---|
| Deep detail about one thing | “What have the critics said about how the Pixies were connected to Nirvana?” | 27 | 1.00 * |
| What came out of what | “What did punk rock evolve into?” | 15 | 0.66 |
| Who or what belongs here | “What are the essential krautrock bands?” | 14 | 0.76 |
| How two things relate | “How does ambient relate to minimalism?” | 12 | 0.92 |
| Where and when | “Where and when did grunge emerge?” | 10 | 0.94 |
| Counting and ranking | “What styles have the most artists?” | 9 | 0.91 |
| Music for a situation | “Music good for getting focus work done” | 7 | 0.96 |
| The path between two things | “What’s the connection between punk rock and dub music?” | 6 | 0.82 |
| Facts about an artist | “Who are the members of the Rolling Stones?” | 5 | 1.00 |
| Facts about the names | “Bands with ethereal, ghostlike names” | 3 | 0.95 |
| Puzzles and constraints | “Which artists have exactly three A’s in their name?” | 3 | 1.00 |
| Straight lookup | “Who is Stromae?” | 2 | 1.00 |
| Category membership | “What are the biggest boy bands?” | 2 | 1.00 |
* The 1.00 on the first row is a ceiling, not a score. Those questions are answered by one section inside a room, and this measurement only checks whether the right room arrived. It cannot see the system picking the wrong paragraph out of the right page. Measured at the section level instead, that row is 46 correct out of 47.
The gold set is 115 questions because it grew by absorbing failures: every play session against the live system pinned the questions it got wrong, so the set is biased toward what the system gets wrong.
Held out (added 2026-08-26). That bias is now measured against three published sets that nobody here wrote, with the predictions filed before the runs. On 610 questions from ArtistMus, hit@10 was 0.97; on 2,085 from Mintaka, 0.85. The third, 900 real recommendation requests from Reddit, scored 0.05 on the first run, half of that the harness’s own fault, and working out why turned up a real defect: asked to recommend artists, the system was answering with a list of genres. The numbers, the predictions and the misses are in Someone else’s ruler.
Next in the series: Someone else's ruler →
Siegfried Martens, 14 August 2026. Part of the Daedalus notes: what was built, what it measured, and what the measurement could not see. Questions and corrections to daedalus@s-martens.com.