By Aaron Tay, Head, Data Services
When conducting research, it is imperative to ensure that your Boolean search strategy is both comprehensive and precise.
But how do you know if your proposed Boolean Search Strategy is effective with high recall (such that it finds as many relevant papers as possible)?
In this Research Radar piece, I will provide a way to estimate how effective your proposed search is.
The obvious course of action, of course, is to test your proposed search strategy against a predetermined set of known relevant papers, often referred to as "gold standard papers," to evaluate its effectiveness and identify areas for improvement.
What Are Gold Standard Papers?
Gold standard papers are articles or documents deemed definitive for the topic you are researching. They serve as benchmarks, as they are either highly cited, recommended by experts, or central to the field you are studying. These papers represent the type of information you do not want to overlook in your search results.
Why Test Against Gold Standards?
Testing your Boolean search strategy against gold standard papers helps ensure that your search terms and logic are capturing all critical articles. If the search misses these papers, it may indicate holes in your strategy, such as missing synonyms, incomplete concept blocks, or overly restrictive Boolean operators.
It is extremely easy to underestimate how tricky it is to construct a comprehensive Boolean Search Strategy. In my experience, whether working with researchers with domain knowledge or independently, testing against gold standard papers will almost always help pinpoint mistaken assumptions and simple blunders that result in the initial search strategy missing many gold standard papers.
Steps to Test Your Boolean Search Strategy
Step 1: Collect Gold Standard Papers
Before testing your Boolean search strategy, you obviously need to compile a list of gold standard papers. How do you do that? Any of the usual methods will suffice.
- Ask for recommendations from domain experts or professors.
- Do preliminary keyword searches in Google Scholar and databases
- Check citation numbers for papers widely referenced in your field.
These days with the help of academic agentic search or deep search tools such as Undermind.ai (SMU has a subscription -- register with your SMU email), Ai2 PaperFinder (free), you can run those and use the top few relevant papers as a Gold Standard. They may not be comprehensive but will help.
Why not then use these tools outright? Firstly, as mentioned these tools may not find everything and you may want a more comprehensive search than what they provide. Secondly, certain types of research methods, such as systematic reviews, require that you search using a structured Boolean Search String that is transparent and reproducible.
You do not need to have a comprehensive gold standard list (I would suggest 10-20 as a reasonable number if that many are available), but aim to ensure they are diverse in terms of the aspects of the topic you hope to catch.
Step 2: Run Your Boolean Search Strategy
Using the Boolean search strategy you've developed, run your query in your academic database of choice (e.g., Scopus, PubMed). Ensure that your search includes the proper concept blocks, synonyms, and operators. You can use the method suggested in the earlier Research Radar piece as well.
Step 3: Compare Results to Gold Standard Papers
Check whether the gold standard papers appear in the results of your Boolean search. If any of the gold standard papers are missing, this is a sign that your strategy needs refinement.
Manually checking your search results against the gold standard set can be painful.
When using a single database like Scopus with a search history function, you can speed up the checking to near instant.
Here's how you do it. You need to be logged into Scopus to do this.
1. Check that the gold standard papers exist in Scopus, by entering the title or DOI one by one into the search. If the paper is found, note the DOI or EID (Elsevier articleID).
2. Go to Scopus Advanced Search and enter in the advanced search something like this to match all the gold standard papers. Below, I am using EID to match all 10 papers; you can use title (TITLE) or dois (DOI).
3. Run your proposed Boolean search
4. In the Search history, select your Gold standard search AND NOT your proposed Boolean search
In the best-case scenario, you receive no results, which means your proposed search strategy includes all the gold standard papers. However, if there are a few results remaining, it indicates that some papers in the gold standard set were not captured by your proposed Boolean search strategy. You should examine why this is the case and modify your search strategy to catch these papers.
Step 4: Analyse Missing Papers
For each missing gold standard paper, try to identify why it was not captured by your search. Common reasons include:
- Missing synonyms: The paper uses terminology not included in your concept blocks.
- Abstract/title mismatch: The terms in your Boolean search do not appear in the paper's title or abstract.
- Incorrect Boolean logic: The combination of operators (AND, OR, NOT) may exclude relevant articles.
A trivial example of the first issue might be due to abbreviations. For instance, I once searched only for "AI," forgetting that some papers might spell it out in full as "artificial intelligence" in the title and abstract (and I was using Scopus, which searches only title abstract), leading to the omission of one paper in the gold standard.
A more advanced example was when I searched for the concept of "Topic Modelling" and did not list all the different techniques used to do it (e.g. LDA).
Another common issue I have seen is "over specification", where too many keywords are combined with AND. This is an easy mistake to make, even for domain experts, who may misremember which words or phrases are truly in all relevant papers, as opposed to only in most of them, resulting in missed relevant papers by forcing matches to certain keywords or phrases.
Again, testing against Gold Standard Papers helps guard against this.
It goes without saying that you need to understand the limitations of the search tool you are using. For example, if a Gold Standard paper is not indexed in the search tool, you cannot expect the search to find it, even if your Boolean is correct! Confirm this by searching the title directly in the tool.
Similarly, if the tool you are using is Scopus and matches only title abstract, you should check for matches only there. If it searches full-text like Google Scholar or some library databases, you should adjust your approach accordingly.
Step 5: Adjust Your Boolean Search Strategy
Use the insights from missing papers to refine your search.
- Add synonyms or alternate phrasing to your concept blocks.
- Consider expanding the fields searched (e.g., keywords, title, abstract).
- Reassess your Boolean logic to ensure inclusivity -- you may have to loosen the logic.
One tip is that certain concepts are very hard to capture with keywords for various reasons, and for a truly comprehensive search, you may want to NOT try to match on these concepts.
For example, clinical researchers who use the PICO (Population, Intervention, Comparison, Outcome) framework often only create concept blocks to match the first three elements but do not do so for "Outcome". This is because they find that studies may express outcomes in so many different ways, so it is better not to match by keywords as this risks losing relevant papers.
Step 6: Retest Against Gold Standard Papers
After refining your strategy, run the search again and compare the results to your gold standard papers. Repeat this process until all or most of your gold standard papers are retrieved.
Balancing Recall and Precision
In an ideal scenario, you have infinite time and energy to review as many papers as possible to ensure you do not miss anything relevant; therefore, you would try utilising extremely broad searches.
In reality, however, you almost always have a limited time budget, necessitating a trade-off between recall and precision. If your query becomes too broad, it may yield irrelevant results, overwhelming you. Conversely, if it is too narrow, you risk missing critical papers.
In the next part of this series, I will share a metric to estimate how broad or narrow your search is, along with some rules of thumb regarding the sweet spot to aim at.
Conclusion
Testing your Boolean search strategy against gold standard papers is a critical step in ensuring the effectiveness of your literature review, as it helps you identify gaps in your search strategy.
While there is no way to guarantee that you have found all relevant papers, I cannot stress enough how this technique has improved the quality of my search strategies.
