REPRODUCIBILITY IN THERAPEUTIC RESEARCH
ReiTheR is funded by the Agence Nationale de la Recherche (reference number ANR-17-CE-36-0010-01). Some ancillary projects were supported by Région Bretagne and Fondation pour la Recherche Médicale and are also presented on this website.
Ideally, medical practitioners and health policy analysts feel more confident when the results and conclusions of scientific experiments can be replicated. However, it has been found that many scientific studies are difficult or impossible to replicate or reproduce. In response, scientists have launched various replication initiatives to assess the robustness of published research. For example, in psychology, numerous labs have volunteered to re-run studies, with the methods used by the original researchers : in one examination of 100 experiments only 39 were successfully replicated. The same concerns have been expressed about biomedical research. But while attempts to replicate psychological studies are often easy to implement and feasible, attempts to replicate medical studies (pre-clinical or clinical) are often costly and difficult to run. This is especially true for randomized controlled trials (RCTs) although these studies are typically of major importance, and expense, in informing health practices and health policies.
It is nonetheless possible to explore, in a first approach, whether qualitatively similar results and conclusions can be drawn from an independent re-analysis of RCT data. An empirical analysis suggests that only a small number of re-analyses of RCTs have been published to date; of these, only a minority were conducted by entirely independent authors. Thirty-five per cent of published re-analyses yielded changes in findings that implied conclusions different from those of the original article as to which patients should be treated. Most re-analyses have been conducted for selected studies, including some very disputed studies, for example study 329, a well known study on paroxetine in adolescent depression presenting the drug as safe and effective while the re-analysis demonstrated a lack of efficacy and some serious safety issues.
Data sharing is increasing in medicine and will enable re-analyses of this sort to be run. Numerous pharmaceutical companies have created mechanisms for investigators to access IPD. Although more than 6000 trials are already available to investigators through open data platforms, only 15.5% had been requested by a small number of researchers. Most proposals have focused on non-pre-specified subgroups or predictors of response rather than on validation of study results.
To date, no independent, systematic and large scale re-analysis project has been published in the field of RCTs. We propose to conduct a project of this sort, which is needed 1/ to explore the existence and the extent of a ‘reproducibility crisis’ in biomedicine and 2/ to better inform editors, funders and other stakeholders as to whether their new policies really make science better, more transparent and reproducible.
OUR TEAM
 Florian NaudetFlorian is a psychiatrist, meta-researcher and former post-doctoral fellow at METRICS (the Meta-research Innovation Center at Stanford). He's currently Professor of Therapeutics at Rennes 1 University, France. His research interests are evaluating and developing methodological solutions to assess treatments in patients, primarily but not exclusive in psychiatric research. He has a strong interest in studying research wastes and data-sharing practices. |  Jeanne GabaJeanne is a PhD student involved in the ReiTheR project. Her thesis is focused on the description of funders' data-sharing policies and the inferential reproducibility of randomized controlled trials available on data-sharing platforms. |  Maximilian SiebertMax is a PhD student involved in the ReiTheR project. His research focuses on data-sharing policies of medical journals and the reproducibility of randomized controlled trials that led to marketing authorization of drugs in the European Union. |
|---|---|---|
 Claude PellenClaude is a pharmacy resident at Rennes 1 University, France. He has a special interest in meta-research and supports open science. His research focuses on the evaluation of the impact of randomized controlled trials data-sharing: does it lead to better or faster knowledge? |  Clara LocherClara is a PharmD with a PhD in Oncology: Pharmacology and Therapeutics'. She is currently assistant professor in pharmacology and methodology. Her research interests are quality, transparency and reproducibility of trials studying targeted therapy in oncology. |  Laura CaquelinLaura is a PhD student at Rennes 1 University, working in Rennes's Clinical Investigation Center (CIC Inserm 1414). She is working mainly on quality, transparency and reproducibility of trials studying targeted therapy in oncology. She provided support in various meta-research projects related with the ReiTheR program. |
 Norah AnthonyNorah is a public health MD interested in methods and statistics in epidemiology and clinical research. Her MD thesis was about impact of clinical trials data re-use. |  Damien BergeatDamien is an upper GI surgeon who is interested in the reproducibility of research in the field of surgery. His first project evaluates the promotion of transparency in surgical research by focusing on data sharing practices and their implementation. Other projects are under consideration on the theme of reproducibility, publication kinetics and level of evidence in the surgical field. |  Alexandre ScanffAlexandre is a public health MD, focused on informatics and epidemiology, with a touch of user experience. He is the geek of the team and always has a technical solution to solve the biggest computing problems. |
 Henri GouraudHenri is a master student at Rennes 1 University. He has an interested in studying the output of data sharing platforms, especially in the context of MIPD meta-analyses. |
OUR RESEARCH
FUNDERS
FUNDERS' DATA SHARING POLICIES IN THERAPEUTIC RESEARCH
Objective: To describe commercial and non-commercial funders data sharing policies and to assess the compliance of funded RCTs to the existing data sharing policies.
Protocol: Open Science Framework
Paper: Gaba JF, Siebert M, Dupuy A, Moher D, Naudet F. Funders' data-sharing policies in therapeutic research: A survey of commercial and non-commercial funders. PLoS One 2020; 15(8): e0237464.
FRENCH FUNDERS DATA SHARING POLICY IN THERAPEUTIC RESEARCH
Objective: To describe French clinical trial funders' data sharing policies.
Protocol: Open Science Framework
Paper (in French): Rollando P, Parc C, Naudet F, Gaba JF. [Data sharing policies of clinical trials funders in France]. Therapie 2020; 75(6): 527-36.
JOURNALS
DATA SHARING RECOMMENDATIONS IN JOURNALS AND RANDOMIZED CONTROLLED TRIALS
Objective: To explore the implementation of the International Committee of Medical Journal Editors (ICMJE) data-sharing policy by ICMJE member journals and by affiliated journals declaring they follow the ICMJE recommendations.
Protocol: Open Science Framework
Paper: Siebert M, Gaba JF, Caquelin L, et al. Data-sharing recommendations in biomedical journals and randomised controlled trials: an audit of journals following the ICMJE recommendations. BMJ Open 2020; 10(5): e038887.
RESEARCH TRANSPARENCY PROMOTION BY SURGICAL JOURNALS
Objective: To describe surgical journals' position statement on data-sharing policies (primary objective) and to describe the other features of their research transparency promotion.
Protocol: Open Science Framework
Paper: Lombard N, Gasmi A, Sulpice L, Boudjema K, Naudet F, Bergeat D. Research transparency promotion by surgical journals publishing randomised controlled trials: a survey. Trials 2020; 21(1): 824.
INTENTION TO SHARE AND RE-USE OF DATA FROM RANDOMIZED CONTROLLED TRIALS DURING THE 10 FIRST YEARS OF ANNALS OF INTERNAL MEDICINE DATA SHARING POLICY: SURVEY OF PUBLISHED STUDIES
Objective: To explore The Annals of Internal Medicine data sharing policy’s impact for randomized controlled trials (RCTs) in terms of output from data sharing (i.e. publications re-using the data).
Protocol: Open Science Framework
PrePrint: MedRxiv
Paper: Pellen C, Caquelin L, Jouvance-Le Bail A, Gaba J, Vérin M, Moher D, Ioannidis JPA, Naudet F. Intent to share Annals of Internal Medicine's trial data was not associated with data re-use. Journal of Clinical Epidemiology. 2021.
DATA SHARING AND REANALYSIS OF RANDOMIZED CONTROLLED TRIALS IN LEADING BIOMEDICAL JOURNALS WITH A FULL DATA SHARING POLICY: SURVEY OF STUDIES PUBLISHED IN THE BMJ AND PLOS MEDICINE
Objective: To explore the effectiveness of data sharing by randomized controlled trials (RCTs) in journals with a full data sharing policy and to describe potential difficulties encountered in the process of performing reanalyses of the primary outcomes.
Protocol: Open Science Framework
Paper: Naudet F, Sakarovitch C, Janiaud P, et al. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. Bmj 2018; 360: k400.
HEALTH AUTHORITIES
THE EVIDENCE BASE FOR PSYCHOTROPIC DRUGS APPROVED BY THE EUROPEAN MEDICINES AGENCY: A META-ASSESSMENT OF ALL EUROPEAN PUBLIC ASSESSMENT REPORTS
Objective: To systematically assess the level of evidence for psychotropic drugs approved by the European Medicines Agency (EMA).
Protocol: PROSPERO
Paper: Erhel F, Scanff A, Naudet F. The evidence base for psychotropic drugs approved by the European Medicines Agency: a meta-assessment of all European Public Assessment Reports. Epidemiology and psychiatric sciences 2020; 29: e120.
DATA SHARING AND RE-ANALYSIS FOR PIVOTAL RANDOMISED CONTROLLED TRIALS ASSESSED BY THE EUROPEAN MEDICINES AGENCY -A SURVEY OF EUROPEAN PUBLIC ASSESSMENT REPORTS
Objective: This registered report introduces a cross-sectional study aiming to assess inferential reproducibility (i.e. Individual Patient Data is available and qualitatively similar conclusions can be drawn from a reanalysis of the original trials) for main RCTs assessed by the European Medicine Agency (EMA).
Protocol: Open Science Framework
Paper: Work in progress, in principle acceptance at BMC Medicine
TIVOZANIB IN RENAL CELL CARCINOMA: A SYSTEMATIC REVIEW OF THE EVIDENCE AND ITS DISSEMINATION IN THE SCIENTIFIC LITERATURE
Objective: To critically review the evidence supporting the approval conflicting decisions by the Food and Drug Administration (FDA) and the European Medicines Agency (EMA); and to analyse the dissemination of this evidence in the literature by way of a citation analysis.
Protocol: No protocol registered
Paper: Work in progress
REPOSITORIES AND OTHER INITIATIVES
STATUS OF SHARING OF INDIVIDUAL PARTICIPANT DATA FROM CLINICAL TRIALS, USE OF SHARED DATA AND IMPACT OF RESEARCH OUTPUTS OF SHARING DATA
Objective: To explore the impact of early clinical trial data initiatives in terms of five outcome domains: status of intent to share data, status of data sharing, use of shared data and impact of research outputs from shared data.
Protocol: Open Science Framework
Paper: Work in progress
ATTENTION AND IMPACT OF SHARING INDIVIDUAL PARTICIPANT DATA FROM CLINICAL TRIALS: A PROTOCOL FOR A CROSS SECTIONAL ANALYSIS OF PUBLISHED RE-USE FROM 3 MAJOR REPOSITORIES
Objective: Randomized clinical trials data sharing policies should have an evaluation component. Our main objective was to assess the impact of published re-uses from RCTs using media attention (Altmetric) and citation rates.
Protocol: Open Science Framework
Paper: Anthony N, Pellen C, Ohmann C, Moher D, Naudet F. Social media attention and citations of published outputs from re-use of clinical trial data: a matched comparison with articles published in the same journals. BMC Medical Research Methodology, 2021; volume 21, Article number: 119
INFERENTIAL REPRODUCIBILITY OF THERAPEUTIC RESEARCH: A CROSS-SECTIONAL STUDY OF RANDOMIZED CONTROLLED TRIALS AVAILABLE ON MAJOR DATA-SHARING PLATFORMS
Objective: This registered report introduces a cross-sectional study aiming to assess inferential reproducibility (qualitatively similar conclusions can be drawn from a reanalysis of the original trials) for main RCTs available on major data-sharing platforms.
Protocol: Open Science Framework
Paper: Work in progress, in principle acceptance at Royal Society Open Science
REPRODUCIBILITY
VIBRATION OF EFFECTS FROM DIVERSE INCLUSION/EXCLUSION CRITERIA AND ANALYTICAL CHOICES: 9216 DIFFERENT WAYS TO PERFORM AN INDIRECT COMPARISON META-ANALYSIS
Objective: Different methodological choices such as inclusion/exclusion criteria and analytical models can yield different results and inferences when meta-analyses are performed. We explored the range of such differences, using several methodological choices for indirect comparison meta-analyses to compare nalmefene and naltrexone in the reduction of alcohol consumption as a case study.
Protocol: Open Science Framework
Paper: Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Med 2019; 17(1): 174.
PERFORMING A LARGE NUMBER OF OVERLAPPING META-ANALYSES ASSESSING ACUPUNCTURE EFFICACY FOR SMOKING CESSATION TO EXPLORE VIBRATION OF EFFECTS
Objective: Different methodological choices such as inclusion/exclusion criteria and analytical models can yield different results and inferences when meta-analyses are performed. We are exploring the range of such differences, using several methodological choices for meta-analyses to compare acupuncture and controls in smoking cessation as a case study.
Protocol: Open Science Framework
Paper: Work in progress
VIBRATION OF EFFECTS IN POOLED ANALYSES OF RANDOMIZED CONTROLLED TRIALS COMPARING CANAGLIFLOZIN VERSUS PLACEBO IN TYPE 2 DIABETES MELLITUS
Objective: Different combinations of randomized controlled trials can yield different results and inferences when pooled analyses are performed. We are exploring the range of such differences, using all possible combinations of studies for pooled analyses to compare canaglifozin and placebo in type 2 diabetes mellitus.
Protocol: Open Science Framework
Paper: Work in progress
QUALITY, TRANSPARENCY AND REPRODUCIBILITY OF THE CLINICAL DEVELOPMENT OF KINASE INHIBITORS IN ONCOLOGY
Objective: To describe the characteristics of kinase inhibitor dose escalation trials performed in the oncology field; and to characterize the level of transparency and reproducibility of this literature.
Protocol: Open Science Framework
Paper: Work in progress
CONFLICTS OF INTEREST / SCIENTIFIC INTEGRITY
PUBLICATION BY ASSOCIATION: THE COVID-19 PANDEMIC REVEALS RELATIONSHIPS BETWEEN AUTHORS AND EDITORS
Summary: During the COVID-19 pandemic, the rush to scientific and political judgments on the merits of hydroxychloroquine was fuelled by dubious papers which may have been published because the authors were not independent from the practices of the journals in which they appeared. This example leads us to consider a new type of illegitimate publishing entity, “self-promotion journals” which could be deployed to serve the instrumentalisation of productivity-based metrics, with a ripple effect on decisions about promotion, tenure, and grant funding.
Protocol: Not Applicable for an Essay
PrePrint: MetaArXiv
Paper: Locher C, Moher D, Cristea IA, Naudet F. Publication by association: how the COVID-19 pandemic has shown relationships between authors and editorial board members in the field of infectious diseases. BMJ evidence-based medicine 2021.
‘NEPOTISTIC JOURNALS’: A SURVEY OF BIOMEDICAL JOURNALS
Objective: Convergent analyses in different disciplines support the use of the Percentage of Papers by the Most Prolific author (PPMP) as a red flag to identify journals that can be suspected of questionable editorial practices. We examined whether this index, complemented by the Gini index, could be useful for identifying cases of potential editorial bias, using a large sample of biomedical journals.
Protocol: Open Science Framework
PrePrint: BioRxiv
ASSOCIATION BETWEEN GIFTS FROM PHARMACEUTICAL COMPANIES TO FRENCH GENERAL PRACTITIONERS AND THEIR DRUG PRESCRIBING PATTERNS
Objective: To evaluate the association between gifts from pharmaceutical companies to French general practitioners (GPs) and their drug prescribing patterns.
Protocol (In French): Open Science Framework
Paper: Goupil B, Balusson F, Naudet F, et al. Association between gifts from pharmaceutical companies to French general practitioners and their drug prescribing patterns in 2016: retrospective study using the French Transparency in Healthcare and National Health Data System databases. Bmj 2019; 367: l6015.
CORRELATION BETWEEN GIFTS FROM PHARMACEUTICAL COMPANIES TO FRENCH MEDICAL ACADEMICS AND THEIR SCIENTIFIC PRODUCTIVITY PATTERNS: A RETROSPECTIVE ANALYSIS USING THE FRENCH TRANSPARENCY IN HEALTHCARE DATABASE
Objective: To investigate the relationship between presently used measures of academic success (e.g. h-index) with the potential financial COIs.
Protocol: Open Science Framework, also available in a blog post
Paper: Work in progress
CONTACT
Interested in our research ? To learn more please contact us.
ReITheR / Florian Naudet
Inserm CIC 1414
22 Rue Henri le Guilloux
35000 Rennes