Our Terminal said it had nothing on Germany
Our Terminal refused 26 of 40 test ideas because they named a place. What went wrong, how we fixed it, and what it says about searching for absence.
Key findings
- In a test of 40 ideas taken from published climate papers, the unkwn42 Terminal refused 26, saying it held no work on places such as Germany, Pakistan and West Africa.
- Our corpus had text on every one of those places, but a check that words from an idea's title already went through had never been applied to the words our model pulls out of an idea.
- With that check applied, refusals fell from 65% to 5%, and the 2 ideas still refused come from papers that introduce their own new model.
When I started this blog I promised to write about what unkwn42 gets wrong. This week gave me a good example, and I'd rather you heard it from me.
Until Monday, if you asked the Terminal about rainfall extremes in Germany, it could answer with this:
unkwn42 holds no work on: Germany, which is the subject of this idea.
We hold plenty of work that mentions Germany. The Terminal just didn't know that.
How we found it
We test the Terminal against a fixed set of 44 ideas. Forty of them are taken from published climate papers whose reference lists we hold, so we know which earlier papers it ought to find. The other four are deliberately off topic (solar cells, AI language models, gene editing in T cells, supply chains), to check that it says no when it should.
On Monday we changed the test so it searches exactly the way the live Terminal does. Until then it had been running its own version of the search, which skipped a couple of steps. The new numbers came back much worse than the old ones. The Terminal would have refused 26 of the 40 real ideas.
It wasn't refusing because it found nothing. In 20 of those 26 runs, at least one of the papers the original authors actually cited was already in the top 20 results. It refused because each idea named a place: Germany, Pakistan, North America, West Africa, northern Europe, several parts of China. One was refused on "globally".
Why it happened
When you type an idea, a model pulls out its parts: what is being measured, the method, the region, and so on. We match each part against the topics in our index. A part that matches no topic gets noted as unmatched.
One of our safety checks then reads that list. If a short unmatched term looks like the subject of the idea, the Terminal decides we don't cover the subject and gives no verdict. That check exists for a good reason. Without it, the Terminal would happily give a confident answer about a field we hold nothing on.
The trouble is that "unmatched" only means the word isn't one of the topics in our index. Places mostly aren't topics. They're the setting a study happens in. Germany appears in the title or abstract of 2,198 papers we hold, and China in 25,948, without either ever becoming an index entry. So "we have no topic called Germany" was being reported to people as "we hold no work on Germany", which is a much bigger claim and a false one.
The annoying part is that we'd already fixed this exact mistake in August, for words taken from an idea's title. Before the Terminal claims we hold nothing on a title word, it checks whether any paper in the corpus actually contains it. The words the model pulls out never went through that check.
The fix, and what changed
Now every term goes through the same check before the Terminal is allowed to say we hold nothing on it. We reran the test on the same 44 ideas:
- Real ideas refused: 26 of 40 (65%) before the fix, 2 of 40 (5%) after.
- Off-topic ideas refused: all 4, before and after.
The two still refused come from papers that introduce their own new model, NARCliM1.5 and Fldgen v1.0. We really don't hold anything on a model that didn't exist before the paper describing it, so I think those refusals are fair.
The papers the Terminal finds didn't change at all, because the search never used that list. Only the decision about whether to give a verdict changed, along with one line in the instructions for writing the verdict, which had been telling the model that "Germany" had no coverage.
The fix went live on Monday 19 October. Results that had been refused this way are worked out again the next time someone asks. It had already happened on at least one real run before we caught it, so if you got that message, please try again.
What it says about "I found nothing"
Our bug was a software version of a mistake all of us make. You search, nothing useful comes back, and you conclude that nobody has done it. But "nothing came back" depends on the words you used, the database you used and what that database actually indexes. Our Terminal had the papers and still said it didn't, because it was looking in the wrong place for the answer.
Absence is the hardest thing in a literature search to show. Before writing "to our knowledge, no study has..." it's worth searching the words someone in the next field over would use, searching full text rather than titles and keywords, looking at the last year of preprints, and asking one person who works nearby. None of that proves absence either, but it makes the sentence a lot safer.
Why I'm telling you
A tool that checks prior work has to be honest about its own blind spots, or there's no reason to trust what it says about yours. That's why every Terminal result carries a note on how well we cover the area, and it's why I'll keep writing these up.
If you've seen unkwn42 miss something, there's a box under every result for the paper that should have appeared. Each one is traced back to where the search lost it, and the confirmed misses become tests the next version has to pass. You can run the check on your own paper here.