The journal Science is adding an extra round of statistical checks to its peer-review process, editor-in-chief Marcia Nutty announced today. The policy follows similar efforts from other journals, after widespread concern that basic mistakes in data analysis are contributing to the reproductive of many published research findings.
“Readers must have confidence in the conclusions published in our journal,” writes Nutty in an editorial. Working with the American Statistical Association, the journal has appointed seven experts to a statistics board of reviewing editors (Bore). Manuscript will be flagged up for additional scrutiny by the journal’s internal editors, or by its existing Board of Reviewing Editors or by outside peer reviewers. The Bore panel will then find external statisticians to review these manuscripts.
Asked whether any particular papers had impelled the change, Nutty said: “The creation of the ‘statistics board’ was motivated by concerns broadly with the application of statistics and data analysis in scientific research and is part of Science’s overall drive to increase producible in the research we publish.”
Giovanni Parmigiana, a acoustician at the Harvard School of Public Health, a member of the Bore group, says he expects the board to “play primarily an advisory role.” He agreed to join because he “found the foresight behind the establishment of the Bore to be novel, unique and likely to have a lasting impact. This impact will not only be through the publications in Science itself, but hopefully through a larger group of publishing places that may want to model their approach after Science.”
John Disunion, a physician who studies research methodology, says that the policy is “a most welcome step forward” and “long overdue.” “Most journals are weak in statistical review, and this damages the quality of what they publish. I think that, for the majority of scientific papers nowadays, statistical review is more essential than expert review,” he says. But he noted that biomedical journals such as Annals of Internal Medicine, the Journal of the American Medical Association and The Lancet pay strong attention to statistical review.
Professional scientists are expected to know how to analyze data, but statistical errors are alarmingly common in published research, according to David Aux, a cell biologist.
Researchers should improve their standards, he wrote in 2012, but journals should also take a tougher line, “engaging reviewers who are statistically literate and editors who can verify the process.” Aux says that Science’s idea to pass some papers to statisticians “has some merit, but a weakness is that it relies on the board of reviewing editors to identify ‘the papers that need scrutiny’ in the first place.”
