Philip S Thomas, Erik Learned-Miller and My Phan's contribution to the Discussion of ‘Estimating means of bounded random variables by betting’ by Waudby-Smith and Ramdas

原文摘要

We enthusiastically agree about the importance of developing tighter confidence intervals and sequences for the mean—the topic of this paper. There are increasing concerns about the safety and fairness of machine learning systems, and we believe one powerful tool for ensuring that machine learning systems are safe, fair, and otherwise reliable is for them to provide guarantees in the form of their confidence that the models they produce will satisfy certain safety or fairness guarantees. At the core of such methods are mechanisms for producing confidence intervals on parameters related to fairness and safety. However, often these confidence intervals must be constructed from samples from extreme distributions for which normality assumptions remain unreasonable even with relatively large numbers of samples. In such cases, confidence intervals like those proposed in this paper are critical, because they do not rely on normality assumptions yet remain relatively tight. We appreciate the comparison of the Hedged-CI method to our own recent confidence interval for the mean (‘PTL21’ in the paper), and find the further increase in tightness compelling, as it could result in improved data-efficiency of safe and fair machine learning methods. Furthermore, we find the extension of the proposed methods to the confidence sequence setting compelling, as it could allow for the re-testing of the safety or fairness of machine learning models as additional data becomes available. One additional comparison that would be interesting to see would be to the confidence interval proposed by Gaffke (2005). (Among the three statistics presented by Gaffke, we are referring to statistic ‘K’.) Showing that this confidence interval has (or does not have) guaranteed coverage remains an open question. Nevertheless, given the remarkable tightness of Gaffke’s bound and the substantial effort that has gone into finding distributions for which it does not exhibit coverage (with no success), it seems like a promising direction for future analysis.

摘要来源：OpenAlex 索引摘要（稿件版本未注明）