Sooahn Shin (Harvard)

Title

Triage Score: A Counterfactual Risk Assessment Instrument

Abstract

Risk assessment instruments, or ``risk scores,'' are widely used in high-stakes decision-making settings such as medicine and the criminal justice system. A risk score predicts the likelihood of an undesired outcome if no intervention is made.  Thus, a sufficiently high score is often interpreted as a recommendation to intervene. However, risk scores fail to account for what would happen if a decision-maker does intervene.  This is problematic because effective decision-making requires consideration of how intervention affects outcomes.  We propose ``triage scores,'' which are based on counterfactual utilities.  Unlike risk scores, triage scores incorporate counterfactual outcomes under alternative decisions, enabling decision-makers to incorporate a wide range of ethical and practical factors. We illustrate the use of triage scores with an application to our own randomized controlled trial evaluating a pre-trial risk assessment instrument.  Our analysis demonstrates that triage scores are able to capture richer utility structures than risk scores and yield substantively distinct results regarding policy evaluation and learning.