Jef Akst has posted an interesting article over at The Scientist discussing a new system for scientists to publish their work directly to the web without traditional journals or the peer review system. Radical, to say the least. In this system, once a group believes that their research is ready for public dissemination they can post their work using a system called “liquid publication”. It appears that they are developing a framework for this system which will allow users to search manuscripts, experiments, datasets, and blogs which have been posted within the system. From their description, it isn’t clear to me if the data is hosted on the laboratory’s own website, or if they post it to an assigned space within the LiquidPub project, but either way the material becomes available immediately without peer review or external editing.
The project springs partially from the math and physics fields which have posted the first external drafts of their papers to a website called arXiv.com for years. According to a physicist friend of mine, this website is used to inform the rest of the field of the most recent advances while also establishing provenance of ideas and discoveries. However, in contrast to LiquidPub, articles deposited in arXiv are still published in the traditional system, including peer review, after their posting to the website.
Fabio Casati, the driving force behind LiquidPub, decided to scrap the peer review process entirely. From their press release:
“We’ve studied this and found that peer review doesn’t work, in the sense that there seems to be very little correlation between the judgement of peer reviewers and the fate of a paper after publication,” says Casati. “Many papers get very high marks from their peer reviewers but have little effect on the field. And on the other hand, many papers get average ratings but have a big impact.”
This isn’t a new concept to scientists – getting a publication into print at a ‘prestigious’ journal is good, but the true mark of an important paper is its citation history. For this reason, institutions judge established scientists by a variety of rubrics that hinge on the citation history of their work as a whole, the most common of which is the H-Index. The LiquidPub system simply eliminates the first step, a step which many regard as a flawed system to begin with.
Can we live without peer review?
Ideally, before a paper is disseminated to the public, the most qualified scientists would examine it closely, ensuring that the approaches taken to address the questions asked were appropriate; that the data were collected, processed, and interpreted correctly; and that conclusions drawn from the study were solid and not over-reaching. This process would prevent bad work from being published, and ensure a solid foundation laid for the next course of studies. However, the reality of peer review can fall far short of this ideal. In reality, the most qualified scientists to examine a group’s manuscript are probably executing competing projects, and the publishing group has likely asked that these competitors are not allowed to read the manuscript due to a conflict of interest. On top of that, scientists are busy, and may not review manuscripts as closely as they should. In the past couple of months I have reviewed two manuscripts with fundamental flaws, and in both cases, I was the only reviewer of three to have noticed these problems. I’m sure any of us could create a list of papers within our field in which the reviewers didn’t catch the fundamental flaw in the manuscript and the paper was published. On top of that, there is a growing concern that peer review is being abused by some of the scientific community. In light of all of this, I don’t think it is as crazy as it might first seem to ask if we even need peer review.
The LiquidPub project aims to use a form of endorsement of the work as a substitute for peer review. The way they describe this is that each PI will have their own ‘liquid journal’ in which they can place papers (or datasets, et cetra) that they find ‘worthy’. The statistics of how often a particular paper has been endorsed in such a fashion will presumably be collected to show the quality and impact of the manuscript much faster than can even be seen by following the manuscript’s citation history. In my simple-minded way, this doesn’t seem all that different from the Digg, Stumble, or Reddit rating systems for internet pages.
I foresee a couple of potential problems with this. The first is simply politics – if I am a PI with one or more of these liquid journals, and I don’t click the equivalent of the “Like” button on Dr. Bigshot’s new manuscript, it could have negative ramifications to my career. Therefore, the possibility exists that these endorsements of manuscripts will simply be given blindly to avoid offence, and in the process lose all meaning. Those of us with LinkedIn accounts are already all too familiar with this problem.
The second problem is that traditional peer review isn’t an all-or-nothing decision, and is often an iterative process that can, when executed properly, make a good paper better. Sometimes the reviewers point out some correctable flaws in the work, or provide pressure to tighten controls on a critical experiment. If I understand it correctly, in the LiquidPub system, the work would be archived as is, warts and all, after the group posted it to the website. From what I’m reading, the publishing group could then respond to comments from the community and improve the paper, but these changes would be documented as new versions of the manuscript. To get around this problem, researchers could simply send their manuscripts out to other members of their community and solicit reviews themselves to improve the work before posting it to LiquidPub or an equivalent system. (This approach is already used by members of the National Academy of Sciences who choose to self-edit their manuscripts for publication in PNAS.)
While I see a lot of potential in this system, it is hard for me to imagine the scientific community excepting such a radically different approach to the dissemination of their results. Then again, it is also hard to imagine a community that exists at the frontiers of technology and knowledge continuing to use a publication system that is still largely based on the printing press.