Hey, NASA: this is what peer review actually looks like

When new research in any field is published to great media fanfare, one thing is guaranteed: in departmental coffee rooms around the world, scientists in that field are ripping it to shreds. This is partly because there is a need to understand the implications of the new findings for these scientists’ own research, and the future of their field. It may be that the conclusions of this new study seem to go against these scientists’ own ideas of how their particular part of the world works. And because scientists are human, there’s probably some annoyance that a competitor has got their research through the Nature/Science ‘sexy’ filter’ where others have not, and there is a desire to take them down a peg or two.

So the new paper gets picked apart. Experimental weaknesses and gaps in the data are identified. Better analytical and statistical methods that could have been used, but were not, are identified. Weak links in the chain of argument that leads from data to conclusion are stretched until they break. This is true of any paper, but it’s particularly true of the papers that grace the pages of high-profile journals like Nature and Science. ‘Cutting edge’ also means ‘more likely to be wrong’.

That’s what I consider to be real peer review. The pre-publication stuff is just a quality filter, a check that the paper is not obviously wrong – and an imperfect filter at that. The real test is what happens in the months and years after publication. Sometimes, after further research, the ideas in the paper do stand the test of time, and form a firm foundation for further research in that area. Sometimes it turns out to be wrong, but in interesting ways that increase our understanding of how that little bit of the world works. Sometimes it turns out to be simply wrong.

At this stage, it’s hard to know which of these categories the Science paper describing possible arsenic-utilising bacteria, announced to great fanfare last week, will eventually fall into. But one thing is certain: it is hardly news that other experts in the field are skeptical of the claims in this paper. They’re scientists. This is what scientists do. What is a bit different is that the discussion is taking place in a much more public manner than is usually the case – something that NASA and the authors of the paper don’t seem to like very much. Well, tough cookies. You wanted the publicity. If you’re presenting your research at a NASA press conference in the wake of a firestorm of excitable media speculation, you definitely wanted the publicity. It’s a bit late to claim that you want discussion of your research limited to the peer-reviewed literature.

I’ve actually written before about the real issue here: in this new media world of blogs and twitter streams, it’s much harder to control a story, because other scientists now have the tools to make their criticism just as public as your press releases. Personally, I think it’s great that Rosie Redfield’s critique, amongst others’, is being brought to the public attention. I’ve long maintained that it is only by showing people what a real scientific debate looks like, and how congenitally argumentative scientists are as a tribe, that the climate change/evolution/vaccine ‘debates’ will be shown up for the manipulative shams they are.

It’s no surprise the NASA PR office – and the authors of the paper – might feel differently. In less than a week, their neat little story has been messed up around the edges; they might even feel it’s been hijacked. But burying one’s head in the sand is counterproductive; you should robustly engage the criticisms, just as you would if it were a comment-and-reply in a journal, or a challenging question at a conference.

The new reality is this: if you announce the research in a public venue, the debate should – and increasingly will – take place in that same public venue. The real challenge is how to have these debates – and report them – effectively.