Every system this studio ships gets written up the same way: the numbers it was measured against, the decisions that produced them, and an honest account of what the measurement is blind to. These are working notes rather than marketing, and each one is about something that is live and clickable while you read.
202626 August
What it actually takes to serve a music knowledge graph as flat files, and the four places the approach hurts.
- 43,567 rooms turn out to be 111,728 files and 1.4 GB on disk. Every one is written once by a Python script and then never computed again.
- The cheapest part is the hosting. The expensive parts are a four hour cache that silently hides deploys, and a content delivery network that blocks you from checking your own work.
- One constraint I carried as a launch blocker through two write-ups did not exist. A single look at the account settled it: 56,000 files used of a one million limit.
Read the note →
202626 August
Wikipedia, Wikidata, MusicBrainz and Spotify each know who an artist is. They do not agree, and the interesting part is what you build when you stop trying to make them.
- I never merged the four sources. Every style claim is stored with the source that made it, and agreement is computed afterwards as a separate derived layer that can be recomputed without re-scraping anything.
- A genre called Corrido had 64 artists in it. Thirty-eight were Mexican, none were Spanish, and the entry was for the Spanish dance of that name. Nothing about the data looked wrong.
- The visualization I built to show off the graph is what put artists and geography on the same screen, which is the only reason that kind of error is visible at all.
Read the note →
202626 August
I graded my own system on questions I wrote myself. Then I found three sets written by strangers, filed my predictions before running anything, and watched two of them agree and one of them break.
- On 610 questions from a published benchmark that nobody here wrote, the right room arrived in the top ten 97% of the time. On real recommendation requests from Reddit, 5%.
- The 5% was half a broken measurement and half a real defect. Finding it cost a control worth $0 and fixed a product that was answering “recommend me artists” with a list of genres.
- The worst finding came from rereading one answer, not from a benchmark: a quoted answer claimed a room did not exist, and it did. The cause was a routing step that read one of two names it had correctly found. Three lines, and comparison questions went from fetching what they need 35% of the time to 95%.
Read the note →
202626 August
Four numbers, four wrong conclusions, and the cheap boring checks that caught them. A benchmark tells you something is wrong. It almost never tells you what.
- A benchmark told me my recommendation engine scored 0.05. Counting where the questions went, which cost nothing, showed that 46% of them were being scored against a component that returns nothing when run offline.
- A metric said precision fell after a fix. It had not. The metric was only defined on 14 of 75 cases before and 70 of 75 after, so the two numbers were averages over different populations.
- The most useful control of the day was rereading one answer and not believing it: “we absolutely have a room for that.”
Read the note →
202614 August
How I built a retrieval system for a 43,000-room music knowledge graph, and what happened when the measurement turned out to be wrong six times.
- The evaluation was built on day two, before any retrieval channel existed: recall@10 0.40. Two days after launch, measured by calling the shipped code: 0.89, hit@10 1.00, MRR 0.94, across 115 questions in thirteen shapes.
- Six times the harness measured something other than what ships. Every one of the errors flattered the system.
- What to ask anyone building you a retrieval system: the baseline, the blind spots, and whether the eval calls the code you ship or a copy of it.
Read the note →
Already built, not yet written up
Each of these is a system that is running now. Only the note is missing, so the list is
a table of contents. It carries no dates.
- The agent that audits its own corpus. An agent walking 130 rooms,
inventing its own list of what counts as a defect, for $1.81.
- What strangers actually ask an oracle. Once enough of them have.