Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference
基于大语言模型的 A/B 测试:人类因果推断的替代终点框架
Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost. We study when a treatment effect estimated on LLM outcomes can recover the effect for the human population of interest. Distributional equivalence between LLM and human outcomes would make any standard estimator valid but is unrealistic. We therefore develop a statistical framework…