As AI becomes more advanced, more and more aspects of our daily life are touched by invisible algorithms. However, the more we entrust vital decisions to software, the greater the need becomes to interrogate how they work, and why they reach the conclusions they do.

Concern has been slowly bubbling, with the book 'Weapons of Math Destruction' by Cathy O'Neil highlighting the ways in which these algorithms can influence crucial decision making processes including whether to grant a loan, who to hire, college admissions, and bail decisions. One of the most potent dangers of algorithms is how they incorporate and perpetuate intentional and unintentional bias.


Rachel Bellamy leads the IBM Research Human-Agent Collaboration group which examines, among other things, cognitive bias and how it's coded into AI. She is currently undertaking extensive research into how these algorithms can have the effect of discriminating against certain groups, and how developers can circumnavigate this potential problem.

"The issue is that humans are very bad at decision making," Bellamy says, explaining why AI has been gleefully alighted upon as an alternative to messy human reasoning, under the assumption that it's more logical and accurate. "However, it's clear that AI does suffer from bias itself because AI is based on data and there is bias in that data."

High-profile investigations have discovered glaring problems with the indiscriminate use of such algorithms. For example, a 2016 ProPublica investigation discovered that an algorithm used by US criminal courts was biased against black people, meaning it was more likely to wrongly predict that a black person would re-offend than a white person.

See also: A warning from the past: IBM, the Holocaust, ICE and Silicon Valley

More recently, there has been outrage over the discovery that an algorithm used by Amazon in its hiring process was discriminating against women, downgrading applicants which mentioned things like 'women's chess club' or 'women's college'. Another algorithm that was designed to advise where the company should offer next-day delivery was found to discriminate against poor neighbourhoods that were more likely to be home to higher proportions of black or other ethnic minority inhabitants, leading people to decry the algorithm as 'racist'.

To help ease problems such as these, IBM has created an open source toolkit for developers worried about the biases captured by their machine learning tools.

What is IBM's AI toolkit for developers?

The AI Fairness 360 equips developers with 'metrics for datasets and models to test for biases, explanations for these metrics, and algorithms to mitigate bias in datasets and models', along with tutorials and other research materials.

Bellamy says some of the tools were created by IBM, but many were created by the developer community working in this area: "By being extensible and open, this toolkit is aimed at helping the research community gel around a set of practises for checking bias and also for mitigating bias in machine-learning models."

The group under Bellamy has carried out a number of user studies on helping developers explain why certain predictions were made by machine learning algorithms. 

"Helping people understand the general relationship between how a particular prediction relates to other predictions that have been made and also, more globally, the model is behaving," says Bellamy. 

Some of this research involves examining which factor is most important in predicting a certain outcome. For example, in the case of the bail decision algorithm, certain features such as race were being weighted disproportionately.

See also: Researcher explains how algorithms can create a fairer legal system

The metrics you end up choosing in an algorithm will vary depending on what the model is attempting to ascertain.

"For example, in a loan what you want to do is you probably choose the metric of average odds difference because that metric allows you to see whether you have a lot of false negatives," says Bellamy. "False negatives are not good for your bottom line."

In this case, a false negative refers to when someone is predicted to not pay back a loan but would in actual fact have repaid it, clearly leading to a loss of profit for the company. 

After ascertaining whether - and in what way - your model might be biased, you can attempt to mitigate the bias.

"There are several kinds of bias mitigation options, depending on various factors," says Bellamy. "In cases where it is the data that is causing the bias, you may select an option such as 're-weighing'."

She is careful to say that these are all guidelines at the moment, rather than accepted industry standards. However, the research promotes a more conscious approach to developing and deploying algorithms, something which is sorely needed.