In June 2017, Yale CRIT hosted an international conference titled “Ensuring Safety, Efficacy, and Access to Medical Products in the Age of Global Deregulation.” The following blogpost is the third installment of a blog series with commentaries from the conference participants. The views and opinions expressed in this blogpost are those of the author and do not necessarily reflect the position of Yale CRIT. For more blogposts related to this series, see here or click the tag “YaleCRIT17” below.

Figure. Restoring invisible and abandoned trials (RIAT) icebergs of clinical trials [1].
Image credit: Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. BMJ 2013;346:f2865 (Copyright CC BY-NC 3.0)

As evidence is a very wide concept, I will discuss the evidence base for clinical trials, which remain the most important decisional tool for medical products.

The evidence base for trials is preponderantly made up of studies sponsored by the pharmaceutical industry for industry purposes (registration and commercialization), but no one seems to have the exact figures on percentage of trials by sponsorship.

By and large non-industrial product development and registration is a thing of the past, although state production of the vaccines persists patchily. Past examples are the state development and testing of vaccines against diphtheria and pertussis.

To be of use to decision-makers, evidence must be visible and of sufficient detail and quality.  There are many degrees of visibility of evidence: from brief journal articles to complex regulatory documents, to invisible. Each level of visibility entails differing levels of bias and also ways of interpreting the same set of data due to the amount of detail available. Bias arises because of compression of huge quantities of information into a few pages of a publication [1] and because facts are presented in a certain way, or not at all [2].

The Restoring invisible and abandoned trials (RIAT) icebergs show the different levels of data visibility and granularity [2] The small portion of the RIAT iceberg above the waterline (Figure) is visible because there are either register entries, or published articles or grey literature announcing its existence. The submerged RIAT iceberg on the right of the Figure is completely invisible and no one knows it’s there. However, its gravitational pull distorts reality: you see one iceberg when in fact there are two. In both cases, the lower portion contains a huge amount of detail and data which have not been available even when an iceberg is visible. In pharmaceutical trials, this portion is represented by the submissions to regulators, the meat being clinical study reports or CSRs. CSRs are detailed reports, which are thousands of pages long following a standardized format set down by the International Conference on Harmonisation (ICH) [1,3]. This international format ensures that pharmaceutical trials are fully written up and documented in all their constituent parts (the lower part of RIAT iceberg).

A naïve narrative is that accessing CSRs and other regulatory documents (which is now possible from the European regulator EMA, [4]) allows a complete overview of the evidence on a specific product with consequent minimization of reporting bias and (in the research synthesis world that I inhabit), production of a reliable systematic review. Disregarding the difficulties of access which still plague the EMA releases [5], the narrative is naïve on at least two counts. First, CSRs are commercial documents and are written to serve commercial purposes, which may not be aligned with those of public health. Second, there is no equivalent of ICH for non-pharmaceutically sponsored trials. A very recent survey of sixteen U.K. universities shows that only two have legally binding arrangements to make data transparent (in some way), but the policies are weak and not enforced.

So we may be facing the situation where the RIAT icebergs of non-industry-sponsored trials are less visible, and their bulk may be melting away in a dozen rivulets of documents, diaries, notes, and files which are not collected into one master file and are thus likely to get lost in the depths of the sea. 

These facts currently undermine rational and ethical medical decision making.

Tom Jefferson is a Senior Fellow and a Senior Associate Tutor at the Centre for Evidence-Based Medicine (CEBM) at the University of Oxford. He has engaged in extensive research as part of the Cochrane Collaboration and serves on the editorial team of Cochrane’s Acute Respiratory Infections (ARI) group.

1. Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T. Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 2013;346 doi: http://dx.doi.org/10.1136/bmj.f2865
2. Doshi P, Jefferson T. Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports. BMJ Open 2013 Feb 26;3(2):e002496.
3. International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH): Structure and Content of Clinical Study Reports E3. Current step 4 version dated 30 November 1995. Last accessed 24th August 2017. Available at: http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E3/E3_Guideline.pdf.
4. European Medicines Agency (EMA) Policy on publication of clinical data for medicinal products for human use. Policy/0070. 2 October 2014 EMA/240810/2013. Last accessed 8th Feburary 2017. Available at: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2014/10/WC500174796.pdf.
5. Doshi P, Jefferson T. Open data 5 years on: a case series of 12 freedom of information requests for regulatory data to the European Medicines Agency. Trials 2016;17(1):78.