We hypothesize in our projects that re-analyses of a random sample of unselected Randomised Controlled Trials will result in a good inferential reproducibility. However targeted re-analyses of suspect or misreported trials may lead to changes in findings implying conclusions different from those of the original article. The best example was probably study 329.
Members of our group have perhaps found a good candidate for such a targeted re-analysis, the BACLOVILLE trial recently published in the journal Addiction.
Context: Details about the Baclofen saga can be found in a previous paper: "The use of baclofen for AUD began a decade ago when a cardiologist, Olivier Ameisen, published a book, “Le Dernier Verre” [The Last Glass], reporting his own experience with baclofen in withdrawing from alcohol use. Following this publication, many general practitioners and psychiatrists endorsed off-label use.
2014: The French Agency granted a Temporary Use Recommendation (TUR) to try and oversee the widespread off-label use of baclofen in France. The TUR was a temporary policy pending results of two adequate sized French trials but it was renewed twice in 2017."
2018: The French Agency assessed the two pivotal trials. The first one was negative and the second one was BACLOVILLE. This last study was registered on May 23, 2012 and completed in October, 2015. When submitted to the French Agency, the primary outcome of the BACLOVILLE study was not the one registered in the protocol and the publication in a peer review journal was still awaited. We therefore detailed our concerns and called for an independent restauration and re-analysis of the study. Our concerns have not been adequately addressed in the past.
2019 (13-Dec): BACLOVILLE was published in Addiction. The conclusion highlighted the efficacy of the drug despite some safety issues: "Baclofen was more effective than placebo in reducing alcohol consumption to low‐risk levels. The number of adverse events and more serious adverse events was greater with baclofen than placebo". Addiction has some publishing standards and requires, in an open science spirit, that the protocol is published as a supplementary material. We looked at these documents carefully and had major concerns that potentially invalidates the study.
2020 (13-Jan): We submitted a detailed letter (≈ 1700 words) describing all our concerns including serious deviations between the published paper and its protocol, discrepancies with the French Agency's evaluation and undisclosed conflicts of interests.
2020 (27-Apr): the letter was accepted after nearly 4 months, and no change was required by the editor.
Our concerns were :
- There was a change in the primary outcome, as mentioned in the appendix "Changes in primary outcome after trial commencement", with dramatic implication for the study reliability;
- The appropriate handling of the pre-registered primary outcome is now described as a secondary outcome; - No report of the blinding assessment planned in the protocol despite a serious risk of unblinding;
- An inaccurate presentation of safety data;
- The publication stated that none of the death (twice more in the baclofen group) was related to baclofen while the French Agency said the opposite ;
- Non verifiable statement about an anonymous donor ; - The sponsor sold the data to the company marketing baclofen (Ethypharm) and this was not declared in the paper;
- Were the patient informed that their data was sold ?
- Error in the protocol a time stamped document (the statistical team was not the one who was in charge of the study at this time);
We called for full transparency, including time stamped history of amendments and important events in the conduct of the trial.
2020 (4 may): the letter was published with a response. While we called for full transparency the response was one page long. Here is a cynical summary of this response:
- Concerning change in the primary outcome : there was no change ;
- Concerning the new secondary outcome, it was added during the peer review process: the result observed on this outcome is compatible with both efficacy and inefficacy ; - Concerning the fact that the blinding assessment was not described: not a single word about that;
- Concerning the inaccurate presentation of safety data: there was "not enough space" to be accurate;
- Concerning deaths: "No imputability analysis was conducted"... however, the publication stated that none of the death was related to baclofen. This was in fact based on the investigator judgment. We were not aware of the analysis performed by Ethypharm and submitted to the ANSM (The French National Agency for the Safety of Medicines and Health Products). This is curious as we wrote our concerns in The Lancet psychiatry
and got a reply by The Cagliari group, a group including the last author.
- Concerning the anonymous donor: only one of the team knows his name;
- Concerning the sponsor who sold the data: they agree, data was sold by the sponsor, but of course not by the investigator. No correction was made to the COI statement ;
- Were the patient informed that their data were sold ? No answer.
- Concerning the protocol with an error: it was a translation error (of course it is frequent to translate a name to another one...).
The study definitely deserves a careful re-analysis. However, when we requested the data, the sponsor answered that data was not available. There is no data-sharing statement in the paper. In such a case, a policy enforcing data-sharing may surely help.