Quantifying and Correcting Measurement Error in LLM-Generated Classifications: A Beta-Binomial Latent Class Model for Correlated Repeated Judgments
Large language models are increasingly used to classify text at scale. Because their outputs are stochastic, it is common to query a model several times per item and aggregate the results, and latent class models offer a principled way to convert such repeated ratings into estimates of outcome prevalence and of the model's error rates. These models assume that repeated ratings of the same item are conditionally independent given the true label. We examine this assumption…