"Registered Reports is a publishing format that emphasizes the importance of the research question and the quality of methodology by conducting peer review prior to data collection. High quality protocols are then provisionally accepted for publication if the authors follow through with the registered methodology.
This format is designed to reward best practices in adhering to the hypothetico-deductive model of the scientific method. It eliminates a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to report serendipitous findings." [From the Center for Open Science, see here]
We had two registered reports about our reproducibility checks. The first one is now published at BMC Medicine and focused on trials used for regulatory purposes. The second one received in principle acceptance at Royal Society Open Science (RSOS). It is more general and focused on trials sharing data on various data sharing platforms.
Choosing the registered report format
Those two researches were part of two PhD works performed in collaboration and in parallel. I remember when we (Jeanne, Max & I) discussed the idea of doing two registered reports for those two papers:
There is a lack of journals accepting registered reports in biomedicine. It was challenging to identify one. BMC Medicine is perhaps the only one. We also tried PLOS Biology, but the editors were not interested. RSOS was a good place for the second RR as it was on the broader topic of reproducibility.Therefore, we had to choose a journal based on the fact that it accepts RRs rather than on its impact factor. But JIF is currently used for hiring, promotion and tenure while registered reports are not (yet) [by the way, it is one thing to endorse DORA and the Hong Kong principles. It is another thing to behave in accordance, especially in our current system of incentives]. We balanced the pro and cons and both Jeanne & Max said that they were willing to choose the registered report format. I was excited to try this new format and thankful that Jeanne & Max didn’t care that much about JIF. While both journals are leading journals with high standards, BMC Medicine has a moderate JIF and RSOS a smaller one.
Pros and cons
For our projects, we gained credibility by having peer review at first. When we asked data to a sponsor we could argue that our protocol was independently reviewed before the data request. It also allowed us to balance the reason « lack is scientific merit » that was put forward by certain sponsors as a reason for not sharing data… because merit was assessed before conducting the research.
Concerning the two projects, the peer review process before starting the research was good. It allowed to clarify certain points and to clarify our process. Therefore, it was good and important to have it before starting our research activities.
However, the peer review process was quite long and doing this in the context of a PhD is quite challenging. Therefore, we were able to finish the first registered report quite on time (i.e. 6 months after the deadline). Still, we are late for the second registered report and time is running out for the second PhD student. We are doing our best to be on time. But there is a challenge here. Importantly the pandemic didn’t help to manage data sharing requests, and we experienced long time, especially for regulatory purposes (this was from our side). Those reasons are related with our specific project and not the process of working using the registered report format. From our experience, I would say that a big challenge is to be timely. For a 3 years PhD (in France it is 3 years), this is not so easy. This should be wisely planned. To be concrete, the problem we had with time was to have delays at the beginning of our projects and not at the end. And as we were not used to it, it was a change. Next time will be better for sure
Last thoughts
Still, I think that the gain in quality was way superior to the problem of time constraints. Second stage peer review was quite fast for the first registered report and mostly relied on checks by the editor. Max defended his PhD before final acceptance. I’m very happy that the jury acknowledged the value of the work based on the fact that there was an in-principle acceptance and on their own evaluation. In addition, it is super important to know that the paper was not accepted based on the results and on the message it conveys but independently of the observed results. It surely reduces bias and I hope that onetime, this format will be mainstream. Therefore, we are looking forward to finishing the second one. Jeanne will surely have to defend her thesis without all the results (of course she has performed other works).
Last, we have another RR with in principle acceptance at BMC Medicine. In a near future I will be able to tell you more about the writing process of RRs in the biomedical field. And hopefully one day, more biomedical journals will offer the RR format…