Gilbert MacKenzie’s contribution to the Discussion of ‘the Discussion Meeting on Probabilistic and statistical aspects of machine learning’

原文摘要

I would like first to congratulate both authors on presenting their very stimulating work in the spirit of the conference’s theme of statisticians and machine learners working in alliance. There is no doubt in my mind that collaboration will lead to better, more parsimonious, models. My comments relate, mainly, to the second discussant’s points on over-parametrization. If a feed-forward neural net (Ripley, 1994) is denoted by Net(x,θ)⁠, where x is the input layer and ϕ is the defining parameter vector, colleagues, in the University of Limerick, cast the problem in a standard non-linear regression mould, thus: where ϵi∼N(0,σ2) (McInerney & Burke, 2022a). This leads to standard Gaussian loge likelihood, ℓ(θ)⁠, for inference (here θ represents the collection of all net parameters to be estimated). Standard theory leads to θ^∼N(θ,Σ) where Σ^=Io(θ^)−1⁠, where Io(θ^) is the observed, positive definite, information matrix, whence the uncertainty in θ can be assessed. Some care is required to avoid the singularities in Σ^ caused by over-fitting, e.g. when dealing with redundant nodes. Given this probability model and its likelihood function, the model space is searched using the Bayesian information criterion. The choice of Bayesian information criterion, rather than the conventional out-of-sample performance criterion for model selection, leads to an increased probability of recovering the true (more parsimonious) model with comparable, or better, out-of-sample performance. A three-stage, backwards, model selection strategy is employed: it first reduces the number of hidden nodes (architectural pruning), then inputs (variable selection), and ends with a final adjustment (fine-tuning). The availability of the variance–covariance matrix opens up opportunities to (a) frame the output of the neural net analysis in a more familiar, regression-like, format and (b) test for the existence of known or hypothetical structures in the data, for example, as in genetic analysis. There are also other modelling opportunities. Analyses carried out to date are promising; in some cases, comparable performance is obtained with half of the parameters required by the standard neural net. This work is at an early stage and the neural nets studied so far are less complicated than those discussed today. However, as the project progresses (McInerney & Burke, 2022b), it aims to cover the neural net piste. Meanwhile, we must not overlook the unresolved problem of ‘explanation’.

摘要来源：OpenAlex 索引摘要（稿件版本未注明）