Out-group homogeneity bias in AI refers to the tendency of AI systems to perceive members of groups outside their training data (out-groups) as being more similar to each other than they actually are. This can lead to inaccurate assessments and unfair results.

It might sound similar to stereotyping bias and confirmation bias since the root cause is biased training data and algorithmic design. However, the difference lies in how AI processes data and generates results based on bias.

How is out-group homogeneity bias different from stereotyping or confirmation biases?

Out-group homogeneity bias fails to see individuality, treating everything as a single homogeneous group. On the other hand, stereotyping applies unfair generalizations and confirmation bias favors the patterns it has learned.

Consider an AI resume screener trained on a dataset of resumes from software engineers. It might excel at evaluating the resumes for technical roles but would struggle with resumes from marketers (out-groups). The AI might undervalue relevant skills and experiences for creative roles overlooking qualified candidates simply because their resumes don't fit the mold it was trained on.

Case study: US Healthcare algorithm

In 2019, a healthcare algorithm used by US hospitals was found to be biased against patients of color.

The algorithm, which was used for over 200 million people, was designed to predict which patients needed extra medical care by analyzing their healthcare cost history. It assumed that cost indicates a person's healthcare needs. However, this assumption did not account for the different ways in which black and white patients pay for healthcare. In a report, it was found that black patients are more likely to pay for active interventions like emergency hospital visits, even if they show signs of uncontrolled illnesses. As a result of this, black patients:

  • Had lower risk scores
  • Were unfairly compared to a healthier population in terms of cost
  • Were not qualified for extra care

This shows how out-group homogeneity bias can create healthcare disparities and can lead to undiagnosed or untreated cases in already disadvantaged populations.