论文检索

通过测度分解实现稀疏线性回归的高效后验采样AI 解读

We consider the problem of sampling from the posterior distribution of a d-dimensional coefficient vector θ, given linear observations y=Xθ+ε. For sparse Bayesian models, such posteriors are in general multimodal, and therefore challenging to sample from. This observation has prompted the exploration of various heuristics that aim at approximating the posterior distribution. In this article, we study a different approach based on decomposing the posterior distribution into a…

最大间隔线性分类器的泛化误差：过参数化下的良性过拟合与高维渐近AI 解读

Modern machine learning classifiers often exhibit vanishing classification error on the training set. They achieve this by learning nonlinear representations of the inputs that map the data into linearly separable classes. Motivated by these phenomena, we revisit high-dimensional maximum margin classification for linearly separable data. We consider a stylized setting in which data (yi,xi), i≤n are i.i.d. with xi∼N(0,Σ) a p-dimensional Gaussian feature vector, and yi∈{+1,−1}…

不受维度限制的岭回归AI 解读

Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we…

相关性调整的去偏 Lasso：协变量模型不准确时的去偏AI 解读

Abstract We consider the problem of estimating a low-dimensional parameter in high-dimensional linear regression. Constructing an approximately unbiased estimate of the parameter of interest is a crucial step towards performing statistical inference. Several authors suggest to orthogonalize both the variable of interest and the outcome with respect to the nuisance variables, and then regress the residual outcome with respect to the residual variable. This is possible if the…

长宽比发散时低秩矩阵估计的基本极限AI 解读

We consider the problem of estimating the factors of a low-rank n×d matrix, when this is corrupted by additive Gaussian noise. A special example of our setting corresponds to clustering mixtures of Gaussians with equal (known) covariances. Simple spectral methods do not take into account the distribution of the entries of these factors and are therefore often suboptimal. Here, we characterize the asymptotics of the minimum estimation error under the assumption that the…

一般高斯设计下的 Lasso 及其假设检验应用AI 解读

The Lasso is a method for high-dimensional regression, which is now commonly used when the number of covariates p is of the same order or larger than the number of observations n. Classical asymptotic normality theory does not apply to this model due to two fundamental reasons: (1) The regularized risk is nonsmooth; (2) The distance between the estimator θˆ and the true parameters vector θ∗ cannot be neglected. As a consequence, standard perturbative arguments that are the…

高维无岭最小二乘插值中的意外现象AI 解读

Interpolators—estimators that achieve zero training error—have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum ℓ2 norm (“ridgeless”) interpolation least squares regression, focusing on the high-dimensional regime in which the number of unknown parameters p is of the same order as the number of samples n. We consider two different models for the feature…

凸惩罚高维回归的根本障碍AI 解读

In high-dimensional regression, we attempt to estimate a parameter vector β0∈Rp from n≲p observations {(yi,xi)}i≤n, where xi∈Rp is a vector of predictors and yi is a response variable. A well-established approach uses convex regularizers to promote specific structures (e.g., sparsity) of the estimate βˆ while allowing for practical algorithms. Theoretical analysis implies that convex penalization schemes have nearly optimal estimation properties in certain settings. However,…

Lasso 的分布：稀疏球上的一致控制与自适应参数调节AI 解读

The Lasso is a popular regression method for high-dimensional problems in which the number of parameters θ1,…,θN, is larger than the number n of samples: N>n. A useful heuristics relates the statistical properties of the Lasso estimator to that of a simple soft-thresholding denoiser, in a denoising problem in which the parameters (θi)i≤N are observed in Gaussian noise, with a carefully tuned variance. Earlier work confirmed this picture in the limit n,N→∞, pointwise in the…

利用近似消息传递估计低秩矩阵AI 解读

Consider the problem of estimating a low-rank matrix when its entries are perturbed by Gaussian noise, a setting that is also known as “spiked model” or “deformed random matrix.” If the empirical distribution of the entries of the spikes is known, optimal estimators that exploit this knowledge can substantially outperform simple spectral approaches. Recent work characterizes the asymptotic accuracy of Bayes-optimal estimators in the high-dimensional limit. In this paper, we…

通过位移凸性分析两层神经网络AI 解读

Fitting a function by using linear combinations of a large number $N$ of “simple” components is one of the most fruitful ideas in statistical learning. This idea lies at the core of a variety of methods, from two-layer neural networks to kernel regression, to boosting. In general, the resulting risk minimization problem is nonconvex and is solved by gradient descent or its variants. Unfortunately, little is known about global convergence properties of these approaches. Here,…

关于“使用 ReLU 深度神经网络进行非参数回归”的讨论AI 解读

We congratulate Johannes Schmidt-Hieber for his elegant and thought-provoking results.His article uses deep-learning-inspired methods in the context of nonparametric regression.Schmidt-Hieber defines a rich class of composition-based functions G(q, d, t, β, K) and a class of sparse multilayer neural networks F(L, p, s, F ).He proves that least squares estimation over the class of sparse neural networks (with suitably chosen architecture (L, p, s, F )) achieves nearly minimax…

通过原型分析进行非负矩阵分解AI 解读

Hamid Javadia* & Andrea Montanarib*a Department of Electrical and Computer Engineering, Rice University, Houston, TX; b Department of Electrical Engineering and Statistics, Stanford University, Stanford, CAColor versions of one or more of the figures in the article can be found online at www.tandfonline.com/r/JASA.Supplementary materials for this article are available online. Please go to www.tandfonline.com/r/JASA.CONTACT Hamid Javadi hrhakim@rice.edu Department of…

非凸损失下经验风险的几何结构AI 解读

Most high-dimensional estimation methods propose to minimize a cost function (empirical risk) that is a sum of losses associated to each data point (each example). In this paper, we focus on the case of nonconvex losses. Classical empirical process theory implies uniform convergence of the empirical (or sample) risk to the population risk. While under additional assumptions, uniform convergence implies consistency of the resulting M-estimator, it does not ensure that the…

Performing statistical inference in high-dimensional models is challenging because of the lack of precise information on the distribution of high-dimensional regularized estimators. Here, we consider linear regression in the high-dimensional regime $p>>n$ and the Lasso estimator: we would like to perform inference on the parameter vector $\theta^{*}\in\mathbb{R}^{p}$. Important progress has been achieved in computing confidence intervals and $p$-values for single coordinates…

在线控制错误发现率与错误发现比例超限的规则AI 解读

Multiple hypothesis testing is a core problem in statistical inference and arises in almost every scientific field. Given a set of null hypotheses $\mathcal{H}(n)=(H_{1},\ldots,H_{n})$, Benjamini and Hochberg [J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 (1995) 289–300] introduced the false discovery rate ($\mathrm{FDR}$), which is the expected proportion of false positives among rejected null hypotheses, and proposed a testing procedure that controls $\mathrm{FDR}$ below a…