67 weeks to review the literature, and still unsure
A full systematic review takes over a year and keeps about 3% of what it finds. Most of us check prior work with far less method and a lot more doubt.
Key findings
- Across 195 completed systematic reviews registered in PROSPERO, the mean time from registration to publication was 67.3 weeks (Borah et al., 2017).
- Those reviews' searches turned up anywhere from 27 to 92,020 studies, and on average 2.94% of what was found ended up in the review.
- Checking a single idea for prior work often happens with no protocol at all, which is why finishing the search so rarely feels like knowing the answer.
During my dissertation I spent months finding papers. I don't think the months were the worst part. The worst part was that at the end of them I still wasn't sure I'd found the ones that mattered.
I wanted to know how that compares with doing it properly, so I looked for numbers on the most careful kind of literature search there is.
What a careful search costs
In 2017, Borah and colleagues looked at 195 systematic reviews of medical interventions that had been registered in PROSPERO, the international register for such reviews, and then published. On average it took 67.3 weeks to get from registration to publication. The reviews had a mean of five authors each.
Their searches turned up anywhere from 27 to 92,020 studies. On average, 2.94% of what a search found made it into the final review. Everything else, around 97 records in every 100, had to be looked at closely enough to be ruled out.
A systematic review is slow on purpose. There's a protocol written before the search starts, records are usually screened by more than one person, and every decision is written down so someone else can check it. It's the best answer we have to the question "what is already known about this?", and it takes well over a year from start to finish, with several people involved.
What most of us do instead
Checking whether an idea has been done, for a thesis chapter or a grant, often happens over an evening or two. A few searches on Google Scholar, a few keyword combinations, a skim of whatever comes back, a pile of open tabs. It's quicker than a systematic review by a long way, and it has none of the safeguards. Nobody writes down which words they tried, and nobody checks the field next door. The search ends because you got tired or the deadline arrived, not because you reached the bottom.
That's why finishing it so rarely feels like knowing. You haven't shown that the work doesn't exist. You've shown that the words you tried, in the database you used, didn't bring it up.
Where the doubt comes from
In my experience, most of it comes from vocabulary. The same idea gets different names in different fields and decades, so the search you ran in your own field's words can't reach the papers written in someone else's. I've written about two examples: marine heatwaves, which were often called warm anomalies before 2016, and yellow substance, which became coloured dissolved organic matter.
The rest comes from volume. If a careful review keeps about 3 in every 100 records it finds, the few that matter to you aren't necessarily on the first page, and a quick search is exactly the kind that stops there.
Asking the smaller question first
Most of the time you don't need a full review to decide whether to start. You need to know whether your specific claim has already been made, and what the nearest work is. That's a much smaller question, and it's worth answering before the big one, because the answer changes how you spend the rest of your reading.
That's the job I'm building unkwn42 for. You describe the idea in plain words, and it comes back with the closest existing work, including papers that use other words for the same thing, a line on how yours differs, and a note on how well we cover your area, so you know when to keep looking elsewhere. It doesn't replace a systematic review, and it isn't meant to. What it answers is the first question, the one that decides whether the weeks after it are well spent.
If you have a paper close to submission, run the check on it. The first one is free.
Sources
- Borah R, Brown AW, Capers PL, Kaiser KA (2017). Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7(2): e012545. doi.org/10.1136/bmjopen-2016-012545