Kentaro Nakamura (Harvard)

Title

Surrogate Representation Inference for Noisy Text Annotations

Abstract

As researchers increasingly rely on machine learning models and LLMs to annotate texts, various approaches have been proposed to correct bias in downstream statistical analysis.  However, existing methods tend to yield large standard errors and require some  error-free human annotation. In this paper, I introduce Surrogate Representation Inference (SRI), which assumes that texts fully mediate the relationship between human annotations and structured variables of interest. The assumption is guaranteed by design provided that human coders rely only on texts for annotation.  Under this setting, I propose a neural network architecture that learns a low-dimensional representation of texts such that the surrogate assumption continues to be satisfied. When multiple human annotations are available, SRI can further correct non-differential measurement errors that may exist in human annotations. Focusing on text-as-outcome settings, I formally establish the identification conditions and semiparametric efficient estimation strategies that enable learning and leveraging such a low-dimensional representation. Simulation studies and a real-world application demonstrate that SRI reduces standard errors by over 50% when machine learning prediction accuracy is moderate and provides valid inference even when human annotations contain non-differential measurement errors.