A signal detection analysis of “dramatic effects” based on recognition heuristics
Recognition is a natural mechanism for making inferences and solving problems – if the object or phenomenon is not recognized, further ascertainment and reasoning processes cannot proceed.34 One strategy that relies on using recognition to make inferences is called the recognition heuristic.35 Recognition heuristic has been demonstrated to provide accurate answers in a wide range of circumstances, particularly when information is limited and uncertain. It is considered to have a special status in our cognitive capabilities because, as explained, if the object is not recognized then it becomes impossible to draw any inferences.35 Therefore, if the regulators do not recognize effect size as a criterion for making decisions on whether to request further RCTs, then effect size would not play a role in their decision-making. However, as discussed, we have demonstrated ecological validity between effect size and decisions whether to require further testing in RCTs: larger effect sizes were associated with a greater likelihood of approval based on nonrandomized data. 15,16 Thus, the magnitude of effect size serves as a recognition heuristic related to the decision to approve drugs based on non-randomized studies. Nonetheless, the specific decision will depend on beliefs (stemming from familiarity) that the effect size exceeding a certain threshold (T ) (e.g., RR>2, 5, 10) is sufficient to obviate testing in further RCTs.
Use of recognition heuristics, like any other decision rule, may result in correct and incorrect inferences. 35 In turn, analyzing the proportion of correct inferences based on recognition heuristics lends itself to inquiry within a framework of SDT.12,36 SDT resides on the notion that the two possible events (signal , e.g. treatment effect is “true”), and noise , e.g. treatment effect is not “true”) have overlapping distributions on a given observation scale. Each of these distributions is further divided into two possible outcomes, which are determined by setting a decision criterion. The criterion divides the signal distribution into true positives (hits, or sensitivity) and false negatives (misses). The noise distribution is composed of true negatives (correct rejections, or specificity) and false positives, respectively.37 To assess the accuracy of recognition heuristic of a continuous variable such as effect size, we assume that judges have a criterion set at one point along the possible values of their prior beliefs, which in our case corresponds to treatment effect size, \(TE=\operatorname{}{(OR}\)). If the TE exceeds the given threshold (T ) consistent with the judges prior beliefs, the rule to activate recognition heuristic can formally be stated as:36
\begin{equation} If\ TE=\ \operatorname{}{OR\geq T}\ ,\ then\ "Approve\ without\ further\ testing\ in\ RCTs"\ \nonumber \\ \end{equation}\begin{equation} If\ TE=\ \operatorname{}{OR<T},\ then\ "\ Approve\ with\ request\ for\ further\ testing\ in\ RCTs"\nonumber \\ \end{equation}
Using the previously defined frameworks for integrating heuristic decision-making with SDT12,36, for each possible cutoff value of \(TE=\operatorname{}{(OR})\), we can calculate standard SDT statistics36,37 including sensitivity, specificity, overall accuracy, positive predictive value (PPV) (i.e.recognition validity ) and d’ (discriminability). In turn, we define the optimal recognition heuristic as the maximum TE threshold for the largest d’ value. [Note that d’(discriminability) represents the standardized distance between the signal i.e., no further RCTs needed and noise (further RCTs required) distributions and is defined as:
\begin{equation} d^{\prime}=\ zHit\ \ zFA,\nonumber \\ \end{equation}
where z Hit and z FA are the z -scores of the true positive and the false alarm rate, respectively].