Bias in data-enabled decisions

Many operational decisions are now heavily reliant on automation - data, models, rules etc.

We’re all trying our best to eliminate bias in the decisions we make as humans.

But we still have a lot of work to do in removing bias from our automated decision support tools.

Legislation in progress

New laws are being enacted or considered, as automated decision support tools gain popularity.

For example, in New York City, automated employment decision support tools must be tested for bias.

This is because these tools can exhibit bias - using personal characteristics in models and rules.

One such set of models and rules relates to hiring.

Because of the sheer volume of applicants that agencies and organisations receive for a given job, automated decision tools are used to screen CVs. In many cases, those tools use some form of AI - for example, a machine learning process/algorithm.

Those are often trained on historical data. So if there were, historically, more successful males than females in job applications, the algorithm may favour male candidates.

This is unfair and perpetuates discrimination.

How can we fix this

One solution to this problem is to remove any gender data that feeds into the model. We don’t tell the automated tool what the applicant’s gender is.

But this can get a bit complicated. The gender field alone is not enough. There are other “proxy indicators” of gender.

For example, a person’s name could suggest their gender. Words like “she” in a CV that’s written in the third person would also be indicative of gender, and ML algorithms can be smart/sophisticated enough to notice this. We need to find all such data and remove it from the model input/features.

Some say that removing these fields is not enough. Importantly, there are also particular situations where removing that data is counterproductive, as described in this article. 

Yes, it's complicated. But, at the core, we want to eliminate bias.

The articles that will follow this one will aim to improve our ability to do that - and achieve fairness.