A Probabilistic Approach to Fairness under Label Bias

TL;DR

We present a framework to audit and learn fair classifiers under label bias by probabilistically reasoning about the expected fairness with respect to hidden fair labels.

Abstract

Machine learning systems are widely used in our daily lives, making fairness an important concern when designing and deploying these systems. Moreover, the data that we use to train or audit them often contain biased labels. In this paper, we show that not only does label bias in training data affect model performance, but it also misrepresents fairness of classifiers at test time. To tackle this problem, we propose a framework to audit and learn fair classifiers by using a probabilistic model to infer the hidden fair labels and estimating the expected fairness under this distribution. In particular, we provide (i) a ``data clean-up’’ method which replaces biased labels with the fair ones—which can be used as pre-processing at train time or for better auditing at test time—and (ii) a reweighting method that directly estimates statistical fairness notions with respect to the inferred fair labels. Experimental results demonstrate the effectiveness of our proposed approach on synthetic data, with controlled ground truth labels and their biased versions, as well as on real-world benchmark datasets.

Citation

@inproceedings{AnchliaTPM23,
  author    = {Anchlia, Saurav and Choi, YooJung},
  title     = {A Probabilistic Approach to Fairness under Label Bias},
  booktitle = {The 6th Workshop on Tractable Probabilistic Modeling (TPM)},
  month     = {aug},
  year      = {2023},
}