Joris Mulder’s contribution to the Discussion of ‘Safe testing’ by Grünwald, de Heide, and Koolen

原文摘要

The potential of e-values (or e-variables) to unite different schools of statistics and allowing optional stopping/continuation in a ‘safe’ manner is a major accomplishment of the authors. The theory implies a change in focus from classical error rates to the expected evidence against the null under the respective hypotheses. Under H0⁠, it is required that the expectation of the evidence against the null, E, should not exceed 1, i.e. EH0[E]≤1⁠, as in formula (1) of ‘Safe testing’. Clearly, using the reciprocal of p-values as a measure of evidence against the null severely violates this criterion as EH0[1/p]=∞⁠. Thus, we see (via a new route) that the routine use of p-values in scientific practice seriously harms the trustworthiness of research findings. Under H1⁠, Bayes factors using a right Haar prior on the scale behave as e-values with an optimal growth rate of the evidence. As stated by the authors, more (nontrivial) e-values exist. As an alternative, I consider fractional Bayes factors (FBFs; O’Hagan, 1995) which are easy to compute and easy to apply as no proper priors need to be formulated. In fractional Bayes methodology, a (minimal) fraction of the data, b, is used to update an improper prior to obtain a proper ‘data-adapted prior’ while the remaining (maximal) fraction is used for computing the evidence (Gilks, 1995; Mulder, 2014). For a simple t-test, the FBF of H1 against H0 is also scale-invariant and given by Γ(nb2)Γ(n−12)Γ(nb−12)Γ(n2)(1+(n−1)−1t2)−n(b−1)/2⁠, where t is the regular t-statistic. For n=20⁠, Figure 1 shows the expected evidence based on a FBF with a minimal fraction of b=2n=0.1⁠, computed with ‘BFpack’ in R (Mulder et al., 2021), and larger fractions, and using a Bayes factor based on the right Haar prior on the scale (Rouder et al., 2009) computed with the ‘BayesFactor’ package (Morey et al., 2024), all on a logarithmic scale. The figure illustrates that FBFs (i) are ‘safe’ as requirement (1) under H0 is satisfied, and (ii) behave competitively to a Bayes factor with optimal growth rate under H1 when using a minimal fraction, with even higher growth rates for large (fixed) nonzero effects. Formal proofs are left for future work. Finally note that for safe sequential testing, FBFs need to be recomputed on the entire (combined) data in order to be coherent. This can simply be done by storing the sufficient statistics or the raw data. Logarithm of the expected evidence E for H1:δ≠0 against H0:δ=0 using FBFs with different fractions b and the Bayes factor with the right Haar prior in case n=20 and varying standardized effects δ. The expected values were computed numerically using 1e7 randomly generated data sets. The horizontal dotted line denotes requirement (1) on a log-scale.

摘要来源：OpenAlex 索引摘要（稿件版本未注明）