Beginning November 1, 2010, The Journal of Neuroscience will no longer allow authors to include supplemental material when they submit new manuscripts and will no longer host supplemental material on its web site for those articles. When articles are published, authors will be allowed to include a footnote with a URL that points to supplemental material on a site they support and maintain, together with a brief description of what the supplemental material includes, but that supplemental material will not be reviewed or hosted by The Journal.
We recognize that this is a major change that will set The Journal apart from most neuroscience journals, but the Society for Neuroscience Council has approved this step because supplemental material has begun to undermine the peer review process in important ways. We believe that the changes described here are our best option for protecting peer review and maintaining our leadership in publishing articles of the greatest significance and highest quality. Because not all of the problems associated with supplemental material will be obvious to readers, we explain them here.
Online supplemental material initially seemed to bring only benefits. Making more information available is a good goal, and the financial costs of storing extra material electronically are small. However, it has become increasing clear that there are other costs to supplemental material as currently implemented. These costs have become obvious as the volume of supplemental material has grown. Although The Journal has published electronically since 1996, supplemental material first appeared around 2003. Since then, the amount of material associated with a typical article has grown dramatically (Fig. 1). While the size of articles has grown gradually over the past decade, the supplemental material associated with a typical Journal article appears to be growing exponentially and is rapidly approaching the size of an article. The sheer volume of supplemental material is adversely affecting peer review.
Average size of a Journal of Neuroscience article and supplemental material in megabytes. Values are trimmed means (5th–95th percentile) to exclude a handful of unaccountably large articles and supplemental files. Supplemental movies are excluded to facilitate comparisons because a megabyte of a movie is arguably easier to evaluate than a megabyte of text, figures, or tables. Data include only articles published in January of each year. Error bars are standard errors of the trimmed means.
Although The Journal, like most journals, currently peer reviews supplemental material, the depth of that review is questionable. Most well qualified reviewers are overburdened with requests to review manuscripts, and many feel that it is too much to ask them to also evaluate supplemental material that can be as extensive as the article itself. It is obvious to editors that most reviewers put far less effort (often no effort) into examining supplemental material. Nevertheless, we certify the supplemental material as having passed peer review.
Another troubling problem associated with supplemental material is that it encourages excessive demands from reviewers. Increasingly, reviewers insist that authors add further analyses or experiments “in the supplemental material.” These additions are invariably subordinate or tangential, but they represent real work for authors and they delay publication. Such requests can be an unjustified burden on authors. In principle, editors can overrule these requests, but this represents additional work for the editors, who may fail to adequately referee this aspect of the review.
Reviewer demands in turn have encouraged authors to respond in a supplemental material arms race. Many authors feel that reviewers have become so demanding they cannot afford to pass up the opportunity to insert any supplemental material that might help immunize them against reviewers' concerns.
Supplemental material also undermines the concept of a self-contained research report by providing a place for critical material to get lost. Methods that are essential for replicating the experiments, analyses that are central to validating the results, and awkward observations are increasingly being relegated to supplemental material. Such material is not supplemental and belongs in the body of the article, but authors can be tempted (or, with some journals, encouraged) to place essential article components in the supplemental material. Precise policies and thorough examination by reviewers and editors might ensure that material is correctly packaged, but this again diverts energy from the review of the primary article, and lapses are inevitable.
We have carefully considered alternatives to removing supplemental material from the peer review process, but have found none acceptable. The idea of demanding that reviewers thoroughly examine supplemental material is impractical. Even if all reviewers could be coerced to review supplemental material with care, it is not clear that this should be encouraged when it would inescapably reduce scrutiny of the main article. Attempting to limit the amount of supplemental material authors can submit is not a solution. Any reasonable fixed limit on what authors can present (e.g., as many figures as are contained in the manuscript) would permit enough material that it would not address the issues of inadequate peer review and misuse by reviewers and authors. Attempting to police submissions so that only important supplemental material was included would leave editors and reviewers with a burden comparable to the one they face now, and one that they are unlikely to take on with greater enthusiasm. Allowing The Journal to host supplemental material that has not been peer reviewed is not an option that the Society for Neuroscience is willing to support.
A change is needed if we are to maintain the integrity and value of peer-reviewed articles. We believe that this is best accomplished by removing the supplemental material from the peer review process and requiring that each submission be evaluated and approved as a complete, self-contained scientific report. By allowing the authors to include a link to supplemental material on their own site, readers will continue to have access to any amount of additional material that the authors consider interesting, but with the clear warning that the material has not gone through peer review. One benefit from this arrangement is that preliminary data presented in this way can subsequently be published without encountering issues of duplicate publication (at least in journals that, like The Journal, do not proscribe prepublication online).
With this change, the review process will focus on whether each manuscript presents important and compelling results. We will of course give clear instructions to our reviewers and editors about our new treatment of supplemental material to ensure that their expectations are appropriate for manuscripts without ancillary material.
Ending The Journal's stewardship of supplemental material is unlikely to cause problems for scientific communication. Perhaps the biggest concern with the new policy is that authors' sites may stop being supported after some period. However, supplemental material is inherently inessential, and we expect that authors will maintain their sites for as long as they consider their supplemental material to be valuable and important. It is conceivable that removing supplemental material from articles might motivate more scientific communities to create repositories for specific types of structured data, which are vastly superior to supplemental material as a mechanism for disseminating data. We should remember that neuroscience thrived for generations without any online supplemental material. Appendices were rare, and used only in situations that seemed well justified, such as presentation of a long derivation.
We recognize that some forms of data, such as videos, can currently only be presented as supplemental material. As we end our support of supplemental material, we will allow authors to publish articles with embedded movies or three-dimensional models, both online and in downloaded PDFs. This change eliminates the only essential role of supplemental material.
We know that our readers recognize the importance of research articles that are self-contained and rigorously peer reviewed, and we hope that they understand the forces that have led us to this change. We similarly hope that our authors and reviewers will appreciate the clarity of having the review process focused on ensuring that each article presents essential findings that are fully described, adequately documented, and free from tangential observations. With your support, the editors reaffirm their commitment to ensuring rigorous and effective peer review so that The Journal maintains its leading position.
