Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation
簇依赖下缺失响应的处理及其在语言模型评估中的应用
Human annotations play a crucial role in evaluating the performance of GenAI models. Two common challenges in practice, however, are missing annotations (the response variable of interest) and cluster dependence among human-AI interactions (e.g., questions asked by the same user may be highly correlated). Reliable inference must address both these issues to achieve unbiased estimation and appropriately quantify uncertainty when estimating average scores from human…