HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
HERO:利用历史数据提高生成模型评估的可靠性与敏感度
Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity. Organizations thus often turn to collecting vast but noisy "silver" labels from crowdsourced workers or vendor annotators as proxies for gold labels. Because gold remains the evaluation target, naively aggregating noisy silver labels may introduce bias, and estimators built on sparsely observed gold…