Explainable AI against the automatization of prejudices
The interpretation of metrics is an important factor in determining fairness or lack thereof. Systematic data quality management and explainable AI are ways that we are tackling prejudice by explaining the concept of fairness: that it applies to sub-groups just as much as to individuals, and that perhaps even certain decisions were made because they were necessary for social progress. In many respects, moreover, explaining decision-making promotes trust in our AI systems. These systems have the potential to revolutionize various fields and industries, but they must be developed and deployed fairly minded and righteously.
Transparency is crucial: The creator of a technical tool is capable of freely designing the algorithm. Decisions about its behavior fall exclusively under the sovereignty of the operator in this context. The behavior of the algorithms of platforms such as Google Internet Search is also partially non-transparent. Providers can change or disable any content. Artificial intelligence adds a new dimension here: while AI methods of the past typically required models specialized for one use case and built on exclusive training data, a generalizing generation of AI is emerging, i.e., programs that excel at multiple tasks simultaneously. For example, modern AIs can summarize a long technical text in a few bullet points and paraphrase a complex piece of content for a different style. Also think of the recently very popular ChatGTP. There is no indication upon which sources given answers are based, if these sources are reliable themselves, and what other potential sources have been neglected.
One way to ensure the fairness of an AI system is to carefully consider the correct interpretation of parameters. It is important to consider not only the overall performance of the AI, but also the performance within specific subgroups that may be susceptible to discrimination. A feature observed in reality with probability p should be ranked by the AI system with the same probability p, not only for the overall data but also within individual subgroups.
To achieve fairness, it is also important to ensure that subgroups have the same ratio of false positives to false negatives. This can be time-consuming because all subgroups must be examined rather than making an overall conclusion. It is also important to consider the impact of misclassified samples, as tolerance for misclassification depends on the application. For example, is it acceptable for 5% of defect-free products to be classified as defective or for 5% of people to be incorrectly denied entry into a country? These questions are especially important when the results of an AI system directly affect a person’s life, such as an AI system used to decide which people to help obtain a new job.
Which tasks are suitable for automatic classification?
AI systems deliver results that go beyond categories like good or bad. An AI for cyclist detection can always be tested for performance, i.e., whether it works appropriately and to the same degree across different people, bikes, environments, groups, and behaviors. It can also be tested to see if it has systematic problems aka bias for certain parts of the application. For example, if it works less well for people with beards and piercings, this bias is easily measurable.
To avoid bias, it is important to identify relevant characteristics and always remove personal information such as age, gender, or nationality. If the performance of the AI system decreases when these features are removed, it may not be a suitable solution for automatic classification. It is also important to make the development and test datasets as diverse as possible to verify that there is no misclassification of individuals or groups.
To raise awareness of the risks of bias introduced by AI systems, it is important to educate customers and stakeholders about these issues. It is also necessary to define fairness for the specific scope of the project and introduce metrics to measure it, such as misclassification rates. It is critical to carefully review the data used to train and test the AI system, in particular by ensuring that it is representative, avoiding the use of features that could lead to discrimination, and verifying that there are no biases or annotation errors.
One approach to assessing and controlling the fairness of an AI system is to use explainable AI and active human control. This involves having a diverse group of testers evaluate the model with different input data, defining evaluation measures for subsets of data, and creating a test set with particularly problematic data or samples. This test set can be enriched with samples from the actual system and comments from users and used to ensure that the model remains fair and unbiased during its development and – if applicable – continuously during operation.
