Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models
尖峰协方差模型中自蒸馏在谱收缩估计量中的最优性
Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We establish that for spiked covariance matrices with $s$ spikes, $s$-step self-distillation achieves optimal performance among spectral shrinkage estimators,…