Type I Error Inflation in Unexpected Event During Survey Designs
The ‘Unexpected Event During Survey' design (UESD) leverages quasi-random timing of interviews relative to external events for causal identification. In this research note, we empirically evaluate the UESD's Type I (false positive) error rate using large-scale survey data covering 40 countries between 2002 and 2024. Based on over 42 million placebo tests, our analysis reveals that standard inferential approaches yield false positives at rates more than double the conventional 5% threshold, indicating miscalibrated p-values and potentially erroneous conclusions. We further document significant heterogeneity in this error inflation across countries, surveys, outcome variables, and model specifications. To address this, we propose a randomization-based adjustment procedure tailored to context-specific false positive rates and provide open-source software for its implementation. We illustrate our approach by replicating the two most highly cited UESD articles published in the APSR.
LOCATION:CGIS Knafel Building, Room K354 STATUS:CONFIRMED DTSTART:20260204T170000Z DTEND:20260204T183000Z END:VEVENT END:VCALENDAR