Andi Wang’s contribution to the Discussion of ‘the Discussion Meeting on Probabilistic and statistical aspects of machine learning’

原文摘要

I would like to congratulate the authors for this excellent article, which includes both careful theoretical developments at the foundations of such denoising models and exciting methodological innovations and experiments. Of course, there are still many unanswered questions in relation to these denoising Markov models. One of the most tantalizing such questions is touched on by the authors in their concluding discussion, namely the nature of the implicit biases which are present in this methodology. It appears to me that there are (at least) two potential sources of implicit bias in such denoising models. First, the use of the evidence lower bound (ELBO) training objective. Although the use of the ELBO is introduced as a concession to the fact that the true likelihood pT(x) is intractable, it has been shown in some cases that using an ELBO actually has beneficial implicit biases. For example, in the context of variational auto-encoders (VAEs), it has been show in certain settings that the use of the ELBO actually enables the VAE to disentangle independent signals (Reizinger et al., 2022). Although for score matching we can see the minimizer is sθ(x,t)=∇xlogqt(x)⁠, I wonder if this particular formulation of the objective leads to some beneficial implicit biases. Second, it is well known in the context of supervised learning that (stochastic) gradient descent often leads to solutions which generalize well (Chizat & Bach, 2020; Soudry et al., 2017)—it is thus natural to ask if a similar phenomenon is occurring in the denoising Markov models setting. Thus one wonders if, in conjunction with the ELBO, the use of gradient descent in this context is resulting in score estimates which ‘generalize’ well, providing novel samples rather than recovering the empirical distribution of the training data. And of course, the particular choice of neural network architecture and network hyperparameters will no doubt further bias the score estimate towards certain approximations. Overall, it would be fascinating to get a handle on which components or combination of components of the methodology are the main contributors towards the very impressive empirical performance. I would like to congratulate the authors again for this fantastic contribution to the literature and heartily agree that further attention is required!

摘要来源：OpenAlex 索引摘要（稿件版本未注明）