The Reproducibility Initiative: Let Them Eat Cake!

Despite obvious differences between the Korean professor-biotechnologist Hwang Woo-suk and German-born postdoc Jan Hendrik Schön, who used to work in the US on semiconductors, both of these scientists have something in common. Since Hwang Woo-suk’s and Schön’s groundbreaking articles were published in Nature and Science, nobody has been able to reproduce their results  and the papers were eventually retracted.

These  withdrawn papers are only a tip of the iceberg – Fanga, et al. estimate, that the number of retracted papers increased tenfold in the last 10 years, while the number of published papers only increased by 44%.

The prevailing culture of “publish or perish”, publishers’ and reviewers’ preference for sleek, perfect “stories” and short term studentships or contracts for researchers put a lot of pressure on scientists, which results in insufficient replication of results, errors and deliberate massaging of figures. Errors in papers in high profile journals are noticed relatively quickly due to the sheer number of people who try and often fail to repeat the results, which increases the chance of whistleblowers. However, most of the irreproducible results remain forever in lab books. On the other hand, groundbreaking results are not easy to publish, unless they have a “brand name” of an eminent scientist in the list of authors.

But modern science is based on trust in published data, and mistrust can unravel the system. Journals already require researchers to submit as much experimental data as possible, which has led to the replacement of “data not shown” by supplemental materials.

To further address this problem, the Reproducibility Initiative was launched in August 2012 by an initiative group, which consists of:

• Science Exchange – a commercial aggregator site of scientific services;
• PLoS ONE – one of the leading open access journals;
• figshare – a non-commercial project, which allows storage and publication of data, including negative results; and
• Mendeley – the free reference manager and academic social network.

The Reproducibility Initiative is designed to help scientists get an important “second opinion”, e.g. independently confirm your results. You submit your material and methods, experimental data and – last but most importantly – money, including a 5% brokerage fee. Then the advisory panel matches you with an anonymous service provider – an institute core facility or a company, which reproduces your results (or not!).  If your data are confirmed, you receive a certificate and can publish the reproduced data in the PLoS ONE Reproducibility Collection and the Figshare Reproducibility Collection repository. The service is confidential, and you may decide whether or not to publish the outcome.

Nature has expressed support for the Initiative – it would be happy if the results reported in its published papers were reproduced independently (2). Ironically, the Nature article talking about this mistakenly states that Rockefeller University Press also endorses the Reproducibility Initiative. This is repudiated in the comments: Rockefeller University Press would be happy to provide a link to the replicated results, but this is as far as it is prepared to go.

However, beyond the initiative group blogs and Nature, it seems that nobody else agrees that paying for voluntary results confirmation by commercial entities is a valid idea, and for many reasons. First of all, money for research is very limited at the moment, so wasting it on doing the job twice instead of using it to get leads for future grants seems extremely unwise. Secondly, the Initiative, at least for now, is not intended for repeating just anybody’s favorite experiments. Instead, it is intended for pre-clinical target validation, which means that you would be sponsoring drug companies’ R&D, while they dismantle their in-house R&D departments. Finally, because the Initiative is voluntary and the results are unenforceable, it’s highly unlikely that deliberate cheats will go through the motions.

I would like to finish with a grandiose but pertinent quote from “The Hydrogen Sonata” by Ian M. Banks:

“Outside of an enforceable legal framework – something that is manifestly not present there –  trust only operates where beings have the concept of honour, and, generally, a reputation – a standing – they want to protect.”

The Reproducibility Initiative as an attempt to buy trust seems to be as futile as the French Empress advice to starving peasants to buy cake when they don’t have bread.

References:

1)    Fanga F.C., Steenc R.G., Cassadevalld A.,Misconduct accounts for the majority of retracted scientific publications. PNAS V. 109 no. 42, pp 17028–17033

2) Baker M.Independent labs to verify high-profile papers. Nature, December 2012