Journal of Clinical Trials
Open Access

Predicting Success of Clinical Trials

Markus Thunecke^{* *}Correspondence: Markus Thunecke, Department of Biochemistry, University of Heidelberg, Catenion GmbH, Munzstrasse 18, Berlin, Germany, Email:

Author info »

Study Description

In spite of huge progress in understanding disease biology and technological advances in patient selection and clinical study design, the failure rates of clinical trials are still very high. According to Hay et al. [1] the probability of drugs in phase III to get approved across different indications was only 50%. This is noteworthy in light of the fact that most drugs that make it into phase III have successfully completed phase II, this implies that they met the primary efficacy endpoint and had an acceptable safety profile. There are several possible reasons for this discrepancy between phase II and III success, sometimes drug company sponsors have such a strong financial and strategic interest to advance projects into phase III that they are willing to accept or even overlook obvious issues and risks. Alternatively, the phase II results may have simply been false positive as phase II studies are not always powered for significance, or the power may have been too low. In some rare cases, phase II may have been skipped altogether, this happens especially in indications where patients are enrolled in phase I, such as oncology. For clinical phases I and II the likelihood of approval is much lower (10.4%, 16.2%) [2]. These issues are exacerbated by the common practice to use adjusted historical success probabilities for go/no go decisions, although such historical averages can at best give a sense of direction. If agency issues and historical data not reflecting the candidates real profile come together, one gets a dangerous mixture leading to risky clinical trials. The effect of a failed phase III can be disastrous for smaller biotech companies (and their investors) but also larger companies are sometimes seriously affected and pushed into merger situations as a consequence of high profile phase III failures. But it is most disappointing for those patients who saw the participation in a clinical study as a real chance of improving their condition.

What is needed is a more objective measure of the real risk of a clinical trial that would allow an early identification of potential winners and to de-select low probability drugs. Companies could improve R&D productivity and corporate valuations and investigators could put their patients on those drugs that have a fair chance of succeeding. It is an interesting ethical question what probability would be acceptable for clinicians and patients to consider participation in a clinical trial.

As we live in the age of big data and machine learning algorithms, it is no surprise that several methods to predict clinical success rates of investigational drugs have been published. The first such study already appeared in 2007 by Schachter et al. who built a Bayesian Model to predict clinical success rates of new chemical entities. This study reported an accuracy of 78% in predicting phase III success of oncology agents. The authors argue that superior selection of drug candidates could improve the industry’s overall economic performance. At that time the employed data sets were still rather small and thus the results of questionable reliability. Still it was an early indicator of the practical feasibility of predicting clinical success rates. Since then, several alternative approaches have been published. Di Masi et al. described a scoring tool to predict likelihood of approval from phase II onwards for oncology compounds [3]. Newer studies often employed neural network and machine learning approaches to make most use of the ever increasing abundance of data on clinical trials both successful and failed. Lo et al. reported a model to predict phase I success of Oncology drugs, the authors claimed that in the test set at year 6 follow up 73% of drugs predicted to be approved were approved [4]. Another study applied machine learning to clinical trials from 2003 to 2015 and reported AUROC values of 0.78 and 0.80 for phase II to approval and phase III to approval predictions [5].

Our own Bayesian model to predict phase III success of oncology drugs reports an AUROC of 73% [6]. We were especially interested in those factors that drive a high probability of success of phase III trials. Perhaps unsurprisingly the two factors that have the highest predictive value are the strength of phase II data and the sponsor’s prior experience in the area. The strength of the phase II data is calculated based on the results of all prior phase II studies and their closeness to the phase III trial in question. It is well known by experienced drug developers that one of the main reasons for phase III failure is changing key parameters versus phase II, such as the precise patient population or the clinical endpoints. The prior experience of a company is also highly relevant, this is reflected in some companies having much higher than average probabilities of success within their areas of focus. In our sample Genentech, Janssen and Celgene stood out, all three highly successful players in oncology. Where a company sits on the learning curve is the result of the richness of its experience reflected in the number of past trials and the success rates of those trials. Past success is a strong indicator of future success, highlighting the importance of experienced drug developers being involved in both decisionmaking (selecting the appropriate drug candidates) and clinical trial design.

The published approaches are being complemented by an increasing number of commercially available databases such as Evaluate Pharma [7] or BioMedTracker [8] that also offer product-specific probabilities of success. The availability of these approaches and tools is greatly benefitting mostly sponsors and investors who subscribe to these data-bases. The limitation is that subscribers do not have access to the actual tool but can only look at pre-evaluated compound probabilities [9].

As the state of play has evolved so rapidly over the last years, one may ask why the impact is still not larger, why are these approaches not (yet) routinely used when it comes to decisionmaking on new drugs, why hasn’t it led to higher probabilities of success? The author’s experience is that many senior decisionmakers within biopharma remain very skeptical as to the validity of these approaches vs. their own experience and intuition. All published approaches so far rely on historical data, mostly split into independent training and validation sets, etc. So from a methodological standpoint these are sound approaches, but without prospective validation, skepticism still prevails. One recent approach has been reported by Zhavoronkov et al. who have predicted outcomes for Novartis’ clinical trials expected to read out in 2020. As no update has been provided it is still unclear how good the predictions really are.

In addition to prospective validation studies, the field also needs tools that support not only project selection and go/no go decisions but the design of clinical studies. A better understanding of the factors that lead to higher risk in clinical trial design would be the first step, then changing those factors to mitigate the risk would be the ultimate goal. This requires models that are not black-box and allow users to understand how individual factors contribute to the success probability of a trial. This is a downside of neural networks as they often lead to black box predictions that are hard to conceptualize, while Bayesian approaches enable an interactive process, thus facilitating conscious learning and decision-making.

References

1. Hay M, Thomas DW, Craighead JL, Economides C, Rosenthal J. Clinical development success rates for investigational drugs. Nat Biotechnol. 2014;32(1):40
2. Schachter AD, Ramoni MF, Baio G, Roberts TG, Finkelstein SN. Economic evaluation of a bayesian model to predict late phase success of new chemical entities. Value Health. 2007;10(5):377-385.
3. Dimasi J, Hermann J, Twyman K, Kondru RK, Stergiopoulos S, Getz KA, et al. A tool for predicting regulatory approval after phase II testing of new oncology compounds. Clin Pharmacol Ther. 2015;98(5):506-513.
4. Beinse, G, Tellier V, Charvet V, Deutsch E, Borget I, Massard C, et al. Prediction of drug approval after phase I clinical trials in oncology resolved. Clinical Can Inform 2019;3:1-10.
5. Lo AW, Siah KW, Wong CH. Machine learning with statistical imputation for predicting drug approvals. 2019.
6. Hegge S, Thunecke M, Krings M, Ruedin L, Mueller JS, von Buenau P. Predicting success of phase III trials in oncology. 2020.
7. Biomedtracker. The drug development process through analysts eyes. 2020.
8. Zhavoronkov A, Kudrin R. Multimodal AI engine for clinical trials outcome prediction: Prospective case study of big pharma for Q2 202. 2020.