Judith ter Schure’s contribution to the Discussion of ‘Safe testing’ by Grünwald, de Heide, and Koolen

原文摘要

Once there was a randomized controlled trial (RCT) that studied the recurrence of stroke. It had assumed for the treatment of interest a 40% reduction in hazard, and had set the sample size accordingly (Kamel et al., 2024). The RCT found a 95% confidence interval of (−55%, 36%) and concluded no statistical significance. ‘Absence of evidence is not evidence of absence’ said the statisticians (Altman & Bland, 1995; Harrell, 2024). But the story goes that the sample size was justified based on an earlier study’s 95% confidence interval of (38%, 68%) (Connolly et al., 2011; Kaul, 2024). In the end, no-one knew whether there was a type-II error guarantee, or whether this result would discourage new similar RCTs. If 40% reduction in hazard had been a smallest effect-size of interest (SESOI), there was evidence against clinical relevance. But 40% is not actually small, and in practice, the use of such worst-case scenarios is rare in RCT power analysis. Setting a SESOI is difficult, especially since it interacts with the fundability of the research because small effects require trials that are impractically large and waste scarce resources (Mandrola, 2024). Hence, a principled use of Neyman–Pearson testing in RCTs is unpopular, which might be more attributed to the practicalities of power than of type-I error. I congratulate the authors for writing a reference paper on optimal e-values in concrete testing problems that also inspires discussing the practicalities of power. Doing so motivates e-values without SESOI-requirements, and comparisons with two-sided Neyman–Pearson tests that also leave an aspect of the alternative hypothesis unspecified at the price of power. ‘Unlimited-horizon’ (ter Schure et al., 2024) and ‘power-one’ (Robbins & Siegmund, 1974) can be emphasized by plotting cumulative power by sample size, for various true-effect/nmax-combinations. Continuously increasing functions visualize the suitable e-value tests, also for the small true-effects. Step functions visualize the Neyman–Pearson tests of fixed-sample size (and extensions to alpha-spending with an nmax)—highlighting the lack of power ‘pay-out’ before nmax in the absence of interim analyses, and complete lack of power increase after nmax where bad combinations are stuck at low power. Suitable e-values allow us to collect data, have a look, and take it from there. In some cases, a worst-case power analysis is impossible. If the question is whether the effect is not smaller than a SESOI, your SESOI specifies the null hypothesis and the worst-case alternative. A parallel situation occurs in non-inferiority testing. This impossibility is not front-and-center, but, in my experience, is apparent from the unpopularity of non-inferiority sample size calculations. No new data were generated or analysed in support of this research.

摘要来源：OpenAlex 索引摘要（稿件版本未注明）