Mini-commentary on BJOG-19-1802.R1 & BJOG-19-1803.R1
Title: To be decoded
Zarko Alfirevic,
Department of Women’s and Children’s Health, University of Liverpool
Simon Gates
Cancer Research UK Clinical Trials Unit, University of Birmingham
Email contact: Zarko Alfirevic, Zarko@liverpool.ac.uk
Randomised trials remain gold standard for evaluation of effectiveness
of medical interventions. However, they are expensive, time consuming
and demand huge efforts from participants, researchers and clinical
services that facilitate them. Even when trials are successfully
completed, peer reviewers often find them wanting. One of the most
common criticisms is a ‘lack of power’ to tackle clinically important
outcomes. Why is this happening time and again?
When an important clinical question creates an equipoise, trialists are
faced with difficult choices. If they design a trial to tackle the most
important (often rare) outcomes aiming to detect modest, but plausible,
risk reductions from a proposed intervention, such studies are rarely
feasible and very expensive forcing most funders to simply walk away.
Common ‘remedies’ are to change the outcome to something more common and
less important (often composite), or to propose an unrealistic risk
reduction, sometimes in excess of 50%. Trials of magnesium sulphate for
neuroprotection included in the seminal Cochrane review (Doyle LW et al.
DOI: 10.1002/14651858.CD004661.pub3) that triggered changes in numerous
guidelines world-wide were not an exception (Table 1).
Cerebral palsy is a rare, but devastating complication of prematurity
and even a very modest reduction would, surely, be worth detecting. The
problem is that even in the most ‘at risk’ groups, the incidence of
cerebral palsy will not exceed 10%. A conventional sample size
calculation estimates that to detect a ‘massive’ 25% risk reduction in
cerebral palsy, in excess of 5,000 women would have to be randomised.
Interestingly, such a priori sample size calculations are not a feature
of most published meta-analyses. Step forward Trial Sequential Analysis
(TSA). The TSA is a deceptively simple concept; meta-analysis sample
size needs to be increased to allow not only for the heterogeneity of
included studies, but also for repeated testing when meta-analyses are
being updated and therefore subjected to repeated significance testing.
Interested readers can find out more from the Copenhagen Trial Unit’s
website - vocal proponents of this methodology
(www.ctu.dk/tsa).
In their two sister papers, Wolf et al have, quite ingeniously, used TSA
to determine the size of their randomised trial and, by doing so,
avoided the risk of their randomised trial and updated meta analysis
being criticised as ‘underpowered’ (Wolf H et al. BJOG 2020 xxxx (RCT);
Wolf H et al. (BJOG 2020 xxxx (SR & MA). Could this concept be a
methodological ‘game changer’ in perinatal trials?
The concept of TSA has been widely criticised, and a Cochrane
Collaboration expert panel recommended against its use
(https://methods.cochrane.org/sites/default/files/public/uploads/tsa_expert_panel_guidance_and_recommendation_final.pdf).
Key criticisms are, first, that decision-makers require a summary of the
currently available evidence and this should not depend on past and
future updates. Second, TSA focuses only on statistical significance.
Interpretation of meta-analysis should be based on the estimates of the
treatment effect and its uncertainty, rather than whether an arbitrary
significance threshold is passed. Third, the ways that evidence
accumulates in systematic reviews and individual trials are
fundamentally different. Review updates are not equivalent to trial
interim analyses; updates are not pre-planned and their number cannot be
determined in advance. Furthermore, reviews address multiple
clinically-relevant effects on several outcomes or subgroup analyses,
which need to be integrated into an overall conclusion. The Cochrane
expert panel concluded: “Any sequential adjustment procedure is
necessarily based on a particular instance of the evolution of evidence
that applies to a limited context and cannot satisfy the requirements of
all decision makers.”
Table 1. Key features of randomised trials of MgSO4given to pregnant women for prevention of cerebral palsy in infants born
preterm