Hans Demetrio Gaebler presents "Overcoming Statistical Challenges in Detecting Discrimination"
Publication information:
Abstract
Outcome tests are a long-standing and widely used approach to detecting discrimination in lending, hiring, policing, and beyond. For example, if White loan recipients are found to default more often than racial minority recipients, the outcome test would suggest that lenders impose a double standard, preferentially lending to riskier White loan applicants. Despite its popularity, outcome tests have long been known to be statistically flawed, sometimes even suggesting discrimination against the group that in reality received preferential treatment. We propose two methods for remedying these statistical shortcomings. First, we show that a twist on standard outcome tests leads to surprisingly strong statistical guarantees. Our test is provably correct under a simple non-parametric assumption that we show — both empirically and theoretically — likely holds in many common scenarios. One limitation of this test is that it is, in some cases, inconclusive. In light of this, we introduce an alternative test of discrimination — which we call risk-adjusted regression — that can handle a broader range of cases, but which requires a richer set of covariates. This latter approach sheds light on the connection between statistical and legal understandings of discrimination.
Full text
Outcome tests are a long-standing and widely used approach to detecting discrimination in lending, hiring, policing, and beyond. For example, if White loan recipients are found to default more often than racial minority recipients, the outcome test would suggest that lenders impose a double standard, preferentially lending to riskier White loan applicants. Despite its popularity, outcome tests have long been known to be statistically flawed, sometimes even suggesting discrimination against the group that in reality received preferential treatment. We propose two methods for remedying these statistical shortcomings. First, we show that a twist on standard outcome tests leads to surprisingly strong statistical guarantees. Our test is provably correct under a simple non-parametric assumption that we show — both empirically and theoretically — likely holds in many common scenarios. One limitation of this test is that it is, in some cases, inconclusive. In light of this, we introduce an alternative test of discrimination — which we call risk-adjusted regression — that can handle a broader range of cases, but which requires a richer set of covariates. This latter approach sheds light on the connection between statistical and legal understandings of discrimination.