Thoughts on Interim Research Products

This post is our response to the NIH Request for Information on Preprints and Interim Research Products in NIH Applications and Reports (see NOT-OD-17-006 for more details)

Our lab lab generates a variety of interim research products to improve the speed of publication, rigor and reproducibility of our work and to make our results widely available to interested parties.

Preprints

We aim to publish our papers as preprints where possible, typically using BioRxiv. We believe that this is an important advance in scientific publishing, allowing us to rapidly publish our work, and gain feedback on our studies from an audience in addition to the peer review process. Our standard lab practice is that publications from our lab are posted to a preprint server either coincident with, or shortly before submission to a peer-reviewed journal. Upon acceptance, a link to the final version of the paper is posted on BioRxiv directing the reader to the canonical article

Datasets

We publish raw datasets along with all analysis scripts for our studies in two venues. We use Github for version control, and at various stages of the publication process, we archive these datasets through Zenodo. This simple process (described here). This has several advantages, including presentation of data not in the final publication, and allows for re-use or statistical re-analysis of our datasets if needed. Our datasets do not include any data that is restricted due to IRB guidance.

Experimental Protocols

Protocols in molecular biology are often described in very limited terms, and can be difficult to repeat without further details lacking in the final publication. We publish our internal laboratory methods at http://bridgeslab.sph.umich.edu/protocols, including version control and revision history to track any changes if our protocols evolve.

Utility of these Products

Preprints have allowed much more rapid access to research data. In my view, this has been especially true for genomics (>1000 papers in BioRxiv) but less so in our area of physiology (38 papers as of this writing). In our field, preprints have also been useful in generating rapid responses to potentially problematic findings in the literature. Examples include http://dx.doi.org/10.1101/061366 and http://dx.doi.org/10.1101/047506. Providing a venue for negative results is important for reducing publication bias.

Preprint submissions have occasionally been challenging, as some journals have ambiguous guidance regarding whether preprint publication precludes future publication. As an example, after consultation with the editor of the American Journal of Physiology - Endocrinology and Metabolism, we found that in spite of guidance on the widely used SHERPA/ROMEO service, preprints were not allowed. We submitted a correction to this service several months ago but it still not updated (or possibly in the interim AJP has changed policies). Our general policy is now to choose journals largely based on whether preprints are acceptable. In terms of comments, we would like more engagement of other scientists in peer review of preprints, but at this time there are no standards in the field, nor are there sufficient incentives to encourage scientists to review preprints, in addition to their normal peer review burden.

Our publications appear to be utilized by the community (400-1300 downloads) and several comments have been obtained either on the article, through social media or via direct email from other scientists. We do not track visits to either our dataset repositories or protocol site at this time.

Thoughts on Citation Standards

Standard guidance of what is required for a citation (permanent URL/DOI, authors, date of publication) would help guide the community towards both utilization of these interim products and understanding what is required to make these products effective.

Potential Impact on Peer Review of NIH Applications

In many cases in my field, demonstrating effectiveness in a technique is an important feasibility consideration. This is best shown by published data using this technique, but especially for junior scientists this can be challenging. Preprints, and public datasets can fill this void and show both that data is being obtained, and that analyses can be done properly. Since both of these can be rapidly made public, this may reduce problems with lag between submission of an application and review.

I believe that reviewers should consider preprints as part of an application, if the applicant so desires, as long as this is relevant to the investigator metric, like other peer-reviewed publications. For the background, innovation and approach sections of the grant, it would be unfair for an applicant to utilize these interim products to bypass page limitations.

In summary, we feel that guidance from the NIH allowing for consideration of interim products would enhance evaluation, rigor and reproducibility of the science they fund, or are considering funding.