Importance of Transparency in Assessing the Feasibility of Modeling Rare Disease

The valuation of drugs for rare diseases is an important and complex issue. Some have questioned whether treatments for rare diseases can be accurately assessed through traditional cost-effectiveness analysis. ¹ Coyle and others ² propose a method for economic evaluation of novel treatments for rare diseases in the article titled “Opportunity Cost of Funding Drugs for Rare Diseases: The Cost-Effectiveness of Eculizumab in Paroxysmal Nocturnal Hemoglobinuria.” We were intrigued by the authors’ conclusions, as we modeled the benefits of eculizumab treatment in the same disease for the manufacturer last year in a submission to policy makers and estimated highly different results. To be clear, we developed this health economic model as paid consultants for Alexion (Alexion did not pay us to write this letter).

As we were unable to obtain the model from the authors, we set out to validate the model ourselves based on two presented estimates that could be used for validation: 11.88 life-years for eculizumab and 10.75 life-years for standard of care (SOC) over 40 years discounted at 5% per year.^2(p7) We created a simple validation model (Figure 1), assuming a constant hazard of mortality and a homogeneous patient population, to assess the implied undiscounted survival curves. Upper and lower confidence intervals were constructed using the estimated incremental discounted life-years presented in Table 2 of Coyle and others.^2(p7)

Figure 1

Actual eculizumab survival from Hillmen and others ³ and Kelly and others ⁴ and implied survival curves in validation model based on Coyle and others’ ² results.

We then reviewed actual event data on survival, including the open-label extension studies described by Hillmen and others ³ comprising 195 paroxysmal nocturnal hemoglobinuria (PNH) patients over 66 months. Hillmen and others^3(p68) reported that “patient survival at 36 months was 97.6% [95% confidence interval (CI): 93.7%–99.1%], which was sustained out to 66 months.” The Leeds cohort is the longest cohort study for PNH patients of which we are aware. The most recent published results are by Kelly and others, ⁴ who included 79 PNH patients treated up to 8 years. Kelly and others^4(p6788) reported that “the 5-year survival rate for the patients treated with eculizumab [was] 95.5% (95% CI: 87.6%–98.5%).” We plotted these survival values relative to those predicted by the model we made that replicated Coyle and others’ ² model (Figure 1). Our estimates of the Coyle model predictions are well outside the confidence intervals of these other studies. Hillmen and others^5(p1254) reported median survival for SOC-treated patients as 10 years; our estimates of the Coyle model predictions for SOC are 62.6% at 10 years.

As the Coyle model consists of 47 distinct health states and at least 34 parameters affecting the transitional probabilities, the ratio of states (and parameters) to patients in the eculizumab clinical trials suggests that many transitional probabilities used in the model are likely dependent on the experience of just a few patients, at best. It is unclear where PNH-specific data for the SOC treated patients would come from, given the near absence of clinical trials evaluating SOC. The parameters for age and gender, important for modeling mortality over 40 years, do not appear in the manuscript or supplemental material. One parameter listed is “Relative rate of thrombosis from taking warfarin as a primary prophylaxis v. not taking warfarin,” which was sourced from an analysis of patients with stage 4 metastatic breast cancer.^2,6 In fact, many sources are not from studies of patients with PNH. The article omits key PNH and eculizumab studies, such as Hillmen and others ³ and Kelly and others, ⁴ from the analysis.

Coyle and others^2(p2) introduce opportunity costs as a necessary consideration for reimbursement decisions, as they represent “the benefits foregone by adopting one intervention over another.” For their opportunity cost argument to be relevant, available resources (i.e., the health care budget) need to be constrained, such that adopting one intervention eliminates the possibility of adopting another (e.g., through increased taxes). However, this is an open issue. The authors reject consideration of a budget impact model for eculizumab in PNH,^2(p11) implying that an avalanche of similar product approvals would become problematic (i.e., “the accumulated impact of similar such decisions will not be negligible”). This reasoning is insufficient, as no evidence is presented on the likely accumulated costs and no argument is made on the threshold where the budget impact justifies rejecting life-saving treatments.

Rare diseases like PNH are generally recognized by health economists and policy makers to be a challenging area for the application of standard health economic assessments. Many experts believe that the application of traditional health economic modeling to orphan drugs can have a discriminatory impact on patients with rare diseases. ⁷ A first step to advance this argument would be to work carefully to build models that make the best use of actual data for the rare disease drugs at issue and then diligently, and transparently, validate them. We question whether the authors have demonstrated that modeling an ultra-orphan disease is feasible, and we caution readers to carefully consider the assumptions made before drawing conclusions from their results.