How do you measure how good you are as a scientist?

How would you compare the impact of two scientists on a field? What if you had to decide which one would get a grant?

Measuring scientific performance is both more complicated and important than it might seem at first. Various methods for measurement and comparison have been proposed, but none of them are perfect.

At first, you might not think that the method for measuring scientific performance doesn’t concern you—because all you care about is doing the best research that you can. However, you should care about this because these metrics are increasingly used by funding bodies and employers to allocate grants and jobs. So, your perceived scientific performance score could seriously impact your career.

Metrics for Measuring Scientific Performance

What are the metrics involved in measuring scientific performance ? The methods that might spring to mind at first are:

  • Recommendations from peers. At first glance, this seems like a good idea in principle. However, it is subject to human nature, so perceived performance will inevitably be affected by personal relationships. Also, if a lesser known scientist publishes a ground-breaking publication, then he/she would likely get less recognition than if the same paper was published by a more eminent colleague.
  • Number of articles published. A long publication list looks good on your CV, but the number of articles published gives no indication of their impact on the field. A small number of publications that have been well-heeded by colleagues in the field (i.e., they are cited often) are better than a long list of publications cited poorly, or not at all.
  • Average number of citations per article published. So, if it’s citations we are interested in, then surely the average number of citations per article is a better number to look at. Well, not really. The average could be skewed greatly by one highly-cited article, so it does not allow a good comparison of overall performance.

The H-Index

In 2005, Jorge E. Hirsch of UCSD published this paper in PNAS in which he put forward the h-index as a metric for measuring and comparing overall scientific productivity of individual scientists.

The h-index has been quickly adopted as the metric of choice for many committees and bodies.

Conceptually, the h-index is pretty simple. You just plot papers versus the number of citations you (or someone else) have received, and the h-index is the number of papers at which the 45-degree line (citations=papers) intercepts the curve, as shown in the diagram below. That is, h equals the number of papers that have received at least h citations. For example, do you have one publication that has been cited at least once? If your answer is yes, then you can go on to your next publication. Have your two publications been cited twice each? If yes, then your h-index is at least two. You can keep going until you get to a “no.”

h-index_plot

So, if you have an h-index of 20, then that means you have 20 papers with at least 20 citations. It also means that you are doing pretty well with your science! Hirsch reckons that after 20 years of research, an h index of 20 is good, 40 is outstanding, and 60 is truly exceptional.

The advantage of the h-index is that it combines productivity (i.e., number of papers produced) and impact (number of citations) in a single number. So, both productivity and impact are required for a high h index; neither a few highly cited papers or a long list of papers with only a handful of (or no!) citations will yield a high h index.

In his paper, Hirsch shows that successful scientists do, indeed, have a high h-index. A simple example is that Nobel prize winners in physics all have high h-indices (84% had an h of at least 30).

Limitations of the H-Index

Although having a single number that measures scientific performance is attractive, the h-index is only a rough indicator of scientific performance and should only be considered as such. Hirsch himself writes:

“Obviously a single number can never give more than a rough approximation to an individual’s multifaceted profile, and many other factors should be considered in combination in evaluating an individual. This and the fact that there can always be exceptions to rules should be kept in mind especially in life-changing decision such as the granting or denying of tenure.”

Limitations of the h-index include:

  • It does not take into account the number of authors on a paper. A scientist who is the sole author of a paper with 100 citations should be given more credit than one who is on a similarly cited paper with ten co-authors.
  • It penalizes early career scientists. Outstanding scientists with only a small number of publications cannot have a high h-index, even if all of those publications and ground-breaking and highly cited. For example, if “Albert Einstein died in early 1906, then his h-index would be stuck at 4 or 5, despite his being widely acknowledged as one of the most important physicists, even considering only his publications to that date.”
  • Review articles have a greater impact on the h-index than original papers since they are generally cited more often.

Calculating the H-Index

There are several online resources you can use to directly calculate a scientist’s h-index. The most established are ISI Web of Knowledge, and Scopus, both of which require a subscription (probably via your institute ), but there are free options too, one of which is Publish or Perish.

If you check your own (or someone else’s) h-index with each of these databases, you might get a different value. This is because each uses a different database to count the total publications and citations. ISI and Scopus use their own databases, and Publish or Perish uses Google Scholar. Each database has different coverage, so will come up with different h-index values. For example, ISI has good coverage of journal publications, but poor coverage of conferences, while Scopus covers conferences better, but has poor journal coverage pre 1992.

The h-index provides a useful metric for scientific performance, but only when viewed in the context of other factors. So, when making decisions that are important to you (funding, job, finding a PI) be sure to read through the publication list, talk to other scientists (and students) and peers, and stage in career. So, keep in mind that an h-index is only one consideration amongst many—and you should definitely know your h-index—but it doesn’t define you (or anyone else) as a scientist.