By Aaron Tay, Head, Data Services
Google Scholar turned 20 recently, and despite being the world's most widely used academic search tool, AI-driven alternatives like Undermind.ai, SciSpace, Scite.ai, Elicit.com etc have risen to challenge it.
Given that SMU has subscriptions to the first three AI academic search tools and the fourth, Elicit.com, has a freemium offering, you might be wondering how they match up to Google Scholar.
In my view, no single tool is best for all situations as they have their distinct strengths and weaknesses, and you should consider using a blend of them depending on your use case.
Below are my suggestions on when to use each tool.
Introduction
Besides Google Scholar, in this post, I will also compare it with the following "AI" academic search tools
- Undermind.ai - agented based academic search
- Elicit.com, SciSpace -- "Semantic search" tools
- Lens.org, OpenAlex -- Keyword based tools, not "AI" but have interesting features
Why these tools? This is because for all their differences, there are all cross-disciplinary, huge "mega" indexes of around 200 million (based on estimates by SearchSmart.org). (Note: Both Elicit.com and Undermind.ai use the Semantic Scholar corpus).
These are much larger than typical academic databases. Even ones such as Scopus or Web of Science (Core collection) are only in the 80 million range (again estimated by SearchSmart.org).
Besides better relevancy in results, AI-enhanced capabilities have brought transformative features to academic discovery, which include
- Answer Generation with Citations: Tools like Scite.ai assistant, SciSpace and Undermind.ai use retrieval-augmented generation (RAG) to directly answer complex research questions with cited references, streamlining the process of synthesizing information.
- Synthesis Table Creation: AI tools like Elicit and SciSpace can extract and organize data into synthesis matrices, significantly reducing the effort required to compare, analyze, and synthesize findings across multiple studies.
This is a complicated and evolving topic, and I will not try to evaluate these features but focus on just the traditional search results.
Using smaller specialised databases to search
Does this mean you should never use smaller specialised databases? Of course not, there are situations where you may want or need to use more specialized databases. See table below.
In fact, for more comprehensive searching, I typically recommend you combine a large mega-index like Google Scholar with a specialized subject database closest to the field you are interested in.
While searching in databases or academic search engines with the largest pools should in theory cover most of what you want, their size and cross disciplinary nature also means sometimes relevant items might not be easily surfaced due to many false hits. This is why smaller or subject specific databases can help ensure things aren't missed, specifically given they typically support use of precise searches using controlled vocabulary.
Google Scholar: The All-Purpose Workhorse for Academic Search
Overview
Google Scholar remains the de facto starting point for most academic researchers. Its sheer scale, ease of use, and extensive cross-disciplinary coverage make it indispensable for general searches, finding seminal works, and exploratory research. However, its keyword-based approach, while robust, lacks the advanced precision features by traditional databases or semantic capabilities offered by newer tools.
Key Features and Strengths
- Unmatched Coverage:
- By far the largest academic database, indexing over 300M+ works, includes full-text access for most major publishers.
- Cross-disciplinary scope ensures it is suitable for virtually any academic field.
- Indexing extends beyond metadata (titles and abstracts) to full-text content, which is critical for finding niche mentions or highly specific terms.
- Highly Cited Papers and Reviews:
- Google Scholar's algorithm prioritises title matches and citation counts, ensuring that seminal and widely referenced papers rise to the top.
- The "review articles" filter simplifies finding comprehensive literature reviews, ideal for starting new projects.
- Versatility:
- Useful features like support of forward citation searching, search alerts and recommendations
- Google Scholar's simplicity and broad applicability make it a go-to tool when researchers are unsure of their exact needs.
Unique Use Cases
- Exploring New Topics:
- Its response time is near instant, making it best for doing multiple searches in quick succession when exploring new areas.
- Full-Text Dependent Queries:
- Its full-text coverage is invaluable for finding niche mentions (e.g., It's the only tool that can reliably find mentions of datasets, search tools or other techniques because it covers the method section in the full-text).
- Example: Searching for use of jargon, tools or specific methodologies (e.g., "Bayesian latent variable modeling" or "Boardex") benefits from Google Scholar's unmatched text coverage.
- Seminal or Foundational Research:
- Its algorithm naturally brings seminal and review papers to the forefront, making it ideal for identifying key studies or highly referenced works on broad topics. Example: For a topic like "AI in education," Google Scholar will surface foundational, highly cited papers and reviews that provide a solid entry point into the subject.
Weaknesses
- Limited Advanced Search Features:
- No support for nested Boolean operators, left or right truncation, or robust field-specific queries (e.g., you cannot filter to matches only in abstracts).
- Search strings capped at 256 characters, limiting complex queries.
- Lacks controlled vocabulary support (e.g., Medical Subject Headings [MeSH]).
- Overwhelming Results:
- Full-text matches can lead to excessive, sometimes irrelevant, results.
- Example: Searching for "Elicit" the search tool yields irrelevant matches unless refined with domain qualifiers like "Elicit.com."
- Systematic Review Limitations:
- Does not support bulk exports of records. Only can see up to 1k results, even manual browsing brings up no more than 1,000 results. Also lacks powerful searcher features to make precise searches (see above).
Alternatives for Power Users
- Lens.org or OpenAlex: One of the closest in size to Google Scholar etc., but offer advanced Boolean logic, truncation, and field-specific queries for precision searching. However, their limited full-text indexing makes them less versatile for specific text-based queries.
Undermind.ai: The Precision-Focused, Iterative AI Search Engine
Overview
Undermind.ai (institutional subscription by SMU) stands out for its use of agent-based iterative search and use of GPT4 as a relevancy evaluator, making it highly effective for detailed, specific queries. By combining keyword and semantic methods with human-like iterative refinement, it mimics the strategies researchers use to try to exhaustively explore a topic.
Key Features
- GPT-4 Relevance Assessment:
- Undermind uses GPT-4 to evaluate the relevance of results based on title, abstract, and metadata, producing results closer to human judgment.
- Example: Queries like "How effective are LLMs in title-abstract screening for systematic reviews?" are parsed semantically by GPT4 for precise relevance.
- Iterative Search Process:
- Adapts and refines searches across multiple rounds, incorporating citation chasing and dynamic adjustments.
- Example: Searching for "Bayesian approaches in dynamic modeling" will refine its focus on relevant subfields through iterations.
- Relatively High Recall:
- Achieves recall rates that typically surpass most traditional tools particularly when query is specific. Example: Using a Systematic review as a gold standard, Undermind had a recall@10 rate of 72.7%, indicating its capability to surface relevant papers early.
- AI generated answers::
- Undermind provides a LLM generated report including a summary and main categories of papers
- "Discuss with expert" function allows you to ask specific questions from papers found
Strengths
- Precision for Specific Queries:
- Best suited for highly detailed or narrow searches where specificity is critical.
- Example: "Evaluate ChatGPT's performance as a screener in systematic reviews using metrics like precision and recall."
- Comprehensive Results:
- Search also leverages citation searching and not just keyword, ensuring seminal and related papers are included.
- Agent-Based Search:
- Operates like a digital research assistant, prompting users to refine queries and adapt strategies.
Weaknesses
- Time-Consuming:
- Queries take 3+ minutes, making it unsuitable for quick searches or for broad exploration where you want to try multiple queries in succession.
- Metadata-Dependent:
- Limited full-text access can hinder performance for deeply text-dependent queries.
- Narrow Scope for Broad Queries:
- Less effective for broad topics like "AI in education", better off using Google Scholar
- It tends to be quite conservative in "making connections" and errs on side of excluding possibly relevant papers or arguably useful papers but do not fit the query exactly.
Best Use Cases
- Scoping of Systematic Reviews: Help with scoping of topics to check if there is a reasonable number of papers on a small, detailed topic
- Specific searching after exploring an area using Google Scholar: Good for focusing on specific areas after doing a broad searcher
- Looking for negative cases: Have a hunch or wondering if a specific type of study exists or has been done but can't find it? Use Undermind to check. Of course, if it fails to find the paper, it doesn't mean the paper doesn't exist, you still have to try other methods (e.g. citation searching, keyword searching) - or ask a librarian to help?
For more details refer to
Elicit.com and SciSpace: Beyond lexical search -- Semantic Search
Overview
These tools leverage semantic search techniques (e.g., matching of dense embeddings or, learned sparse embeddings like SPLADE and other hybrid methods) to match documents by meaning rather than just using exact keyword matches. This allows for natural language queries and broader exploratory capabilities. This class of tools can be used if you are at the start of your research, and you are unsure if you have the right keywords and you want to do frequent successive searches to explore.
Conclusion: Expanding Academic Discovery with Keyword, Semantic, and Citation-Based Tools
The academic search landscape is evolving, offering researchers a suite of powerful tools that go beyond traditional keyword-based methods. While tools like Google Scholar remain indispensable for their extensive full-text coverage and citation weighting, new approaches such as semantic search and advanced citation-based discovery are reshaping how we find and connect with academic knowledge.
Semantic search tools like Elicit.com, SciSpace, and Undermind.ai excel at interpreting queries through meaning and context, uncovering relevant results even when exact keywords are unknown or when natural language queries are used. These tools are especially valuable for exploratory searches or when delving into newer fields with less standardized terminology.
If you would like to find out more on how these tools can help with your research, reach out to us at SMU Libraries.