Using PubCrawler: How to Speed Up Your Literature Searches by Crawling

PubCrawler automatically searches PubMed and GenBank for queries that you specify, and emails you the results, helping you stay on top of your literature searches.

Written by: Neal Melvin

last updated: November 3, 2021

Want to know how PubCrawler can save you valuable time on your literature reviews? Read on…

It often seems like a full-time job searching PubMed for the latest papers on adult neurogenesis, mutations in kinase genes that lead to cancer, or the newest papers published from a major figure in your field. However, falling behind on the latest published research can also be a significant source of stress.

In Search of a Library Assistant

When I was a graduate student, I would diligently search PubMed for the newest articles published on my then-favorite protein, TrkB, and my favorite scientist, Marc Tessier-Lavigne.

However, given the demands of graduate school in the experimental sciences, it can be difficult to keep up with your favorite sub-searches on a regular basis. “If only something existed that would do these periodic searches for me automatically and simply email them to me,” I thought.

Turns out, it existed.

PubCrawler is a free, web-based service that was developed by Karsten Hokamp and Ken Wolfe in the Department of Genetics at Trinity College Dublin, Ireland. It has been helping speed up scientists’ literature searches since 1999.

PubCrawler automatically searches both PubMed and GenBank for text-based queries that you specify, and emails you the results. What’s not to like?!

After signing up for an account, you can customize search queries in several ways, including those offered by PubMed (author, title words, text/abstract keywords, MeSH terms, journal, publication range, etc.).

You also have the option of selecting how frequently your search results are emailed to you and which specific day of the week (and time!) you’d like to receive them.

Although PubCrawler isn’t the prettiest website, looks aren’t everything and it is still an absolutely vital part of me staying on top of the literature in my field. And anything that helps you wade through the mountain of new literature being published weekly is surely a good thing!

How to Use PubCrawler to Go to the Library for You

For example, let’s say that you want to keep abreast of the latest published work from Fred Gage, a prominent player in the field of adult neurogenesis.

Simply construct a new query using “Gage FH” as the search term and use the drop-down menu under “search field” to select “Author”.

Hit the “Finish!” button at the bottom of the page and you’ll now receive emails whenever Fred Gage publishes a new paper according to the frequency and time you’ve set:

Figure 1 pubcrawler
Using PubCrawler: How to Speed Up Your Literature Searches by Crawling

The search queries you can set up with PubCrawler are as diverse as those possible from searching PubMed directly. For example, you might also want to know when papers are published on the role of the NMDA receptor in memory.

For this, simply use “NMDA receptor” as the search term in one box and “memory” as the search terms below.

Figure 3 pubcrawler

Select “and” as the connector to make sure that PubCrawler returns papers that have both of these terms (in this example, eligible returns must have each term in the title or abstract of the paper, but you can restrict it to just the title or broaden it to retrieve papers that have the chosen terms anywhere in the text).

By using combinations of search terms, you can create custom queries to deliver only the papers you want!

Using PubCrawler: How to Speed Up Your Literature Searches by Crawling

A more thorough tutorial can be found here.

If you happen to agree with PubCrawler’s slogan (“It goes to the library – you go to the pub”), then setting up a series of queries is an absolutely vital part of staying on top of the literature in your field in the most efficient way!

I’d suggest using it alongside other RSS feeds as well as eTOCs to ensure you don’t miss a thing.

Are you a fan of PubCrawler? We’d love to hear how it helps you; drop us a comment below!

For more tips on keeping track of the scientific literature, head over to the Bitesize Bio Managing the Scientific Literature Hub.

Originally published July 9, 2016. Reviewed and updated November 2021.

Neal is a Professor at Brandon University Canada with specialties in Stereology, Neurobiology, Neuropsychology, Neuropsychopharmacology, Stem cells, Neurology, Psychiatry, Cell Biology, Molecular biology, Genomics and Genetics.

More 'Taming the Literature' articles