The 20 January issue of Nature has a news article "Trial by Twitter" exploring the issue of how scientists should deal with comments on their papers on the Internet. The article covers a lot of ground but I'd like to deal with a little of it. Also, it is a news article and tries hard to be neutral; I will not be as neutral.
Part of the article touches on two life sciences papers from last year which attracted significant controversy: a genome-wide association study (GWAS) on human longevity and the claim of arsenic-incorporating bacteria. In both cases, the authors initially declined to respond to pointed critiques in blogs. I won't mince words where I stand on this: claiming it is "premature for us to talk about our experience because this is still an ongoing issue" (the GWAS authors) or "any discourse will have to be peer reviewed in the same manner as our paper was" (the arsenic authors) is stonewalling against the interest of science and should be unacceptable in the community of scientists. It is particularly ill advised to make such a statement after, as the arsenic bacterium authors did, engaging explicit courting of the popular press to publicize the work as significant and well-executed.
That said, one cannot expect scientists to know of every tweet or reference to their paper. Nor should scientists have an obligation to address comments which have no basis in science nor engage persons engaged in personal attacks. Obviously that leaves a lot of discretion -- but discretion is different than saying "all roads go through further formal peer review". It is also unacceptable to hide behind the fact that the original paper passed peer review; as a number of papers have demonstrated, peer review often fails for various reasons. The Reactome paper is a particularly painful example of this for me, since I swallowed it whole and later realized that I (and apparently the chosen reviewers) lacked the expertise to spot glaring issues in the paper. But, I do believe that scientists do have an obligation to respond to detailed, reasoned, scientific critiques of their work.
Who should decide which critiques to respond to? The article talks a bit about post-review scientific scoring systems such as the Faculty of 1000. These are useful, but not necessarily the best way to find the most energetic and informed reviewers for a paper. Among those bloggers who comment on a paper, you are likely to find such individuals, though not with a perfect signal-to-noise ratio. Given that traditional journals are constantly struggling to find ways to stay relevant in an Internet world, to me it is the editors at the journal a paper was published in are obvious candidates for selecting those critiques demanding response. Of course, the authors of the papers themselves should also play this role.
The article discusses how poor the response has been to commenting facilities at journal websites which enable such commentary. Now, my one interaction with such a site did lead to a peer-reviewed version of my blog post. But, that is the only time I've actually tried to contact the editors of a paper, and only because I felt so strongly. Editors and authors need to be more proactive, with the expected norm being that they actively look for intelligent commentary on papers they edited or authored. Resources such as Research Blogging can help find these; indeed, I would argue that any editor who doesn't scan Research Blogging for coverage of papers in their journal should immediately start doing so.
The point of scientific publishing and peer review is not to protect reputations and not to promote orthodoxy; the point of these is to attempt to ensure that good science is made better and bad science is swept clean as soon as possible. The existing system of formal review represents an approach to this goal which evolved over time; it is neither perfect nor unquestionable. The time is now for a new ethos in science in which any reasoned source of scientific criticism is accepted and can expect response.
You make some nice points. I like that you're looking at peer review as a means to an end. Unfortunately, a lot of our colleagues have lost that perspective, coming to see the peer review institution itself as the guardian of science.
ReplyDeletePeer review is always going to be (and always has been) a linchpin of science. The form of that review, though, has changed a lot over time, and will change a lot in the future. I share you excitement that peer review aggregating post-publication social media activity can work in the long run.
INteresting Points. However, as you point out, if someone feels strongly about an issue in a published article, they should be either sending in a letter to the editor or contacting the authors directly. When people decide to Blog instead of taking these routes, they have decided to not address the authors directly and because of that, they can't expect a direct response from the authors. Furthermore, let's be honest, we are drowning a in sea of bla bla bla... more articles, more words, more places to post all of this and yet there seems to be less and less fruitful communication (which is, fundamentally, what you saying in your article). I don't think saying that scientist should be responsible for checking all these posts out AND responding to them is a solution (or realistic). Write to the authors, call them, talk to them at a conference... good old fashion communication!
ReplyDelete