EMA and FDA databases of drug approvals based on non-RCT comparisons
To provide empirical support for our theoretical framework, for this analysis we merged the EMA and FDA database of approvals based on non-RCT comparisons15,16, as described above. We merged both databases because of the high concordance rate (91-98%) in approval 26 between the two agencies. We excluded duplicate entries between the two databases, and reviewed all regulatory approval documents that addressed whether testing in subsequent RCTs is required. We included a total of 134 approvals of drugs and devices that were based on non-RCTs. For 35 of these treatments, the regulators required further evidence from RCTs, whereas for 99 treatments they did not. Although this is the best, contemporary dataset available15,16, it is important to note that the agencies often failed to provide explicit comparisons of these drugs and devices against comparators, arguing that in many cases “efficacy has been assessed on the basis of [outcomes] in comparison to what would be expected by expert clinical evaluation and by comparison with previous experience in this type of patient”.15 Indeed, the evaluation of treatment effects always depends on the comparison of experimental (direct or counterfactual) with a control intervention if one is to estimate the effect size. Therefore, when the agencies did not specifically provide comparison data, we imputed the control events either based on our interpretation of the agencies’ judgments documented in the approval reports or the best available data available in the literature.15,16 However, in our attempts to translate the FDA and EMA judgments into the effect sizes, we frequently imputed very low (often equal to zero) event rates such as response rate or survival in the control arm. As a result, we observed some empirically improbable high effect sizes. Nevertheless, our estimates seem to reflect what the agencies often believed – “without new treatments, most patients would surely die15 – implying that these effects are indeed considered self-evidently large, dramatic effects and hence confirming the role of heuristics in the decision-making process of treatment approvals.