As businesses continue to expand and deepen their use of AI, it’s critical that they be able to answer important questions such as, how did an AI model arrive at an insight? Can the results an AI system reaches be explained? Is the technology being used responsibly? As AI is increasingly used to make impactful decisions such as who gets approved for a loan or a health prognosis, being able to answer questions like these is no longer a nice to do, it’s a must-have. This is why explainability is rapidly becoming a critical tool in any business’s AI strategy.
We recently talked about the value (and challenges) of explainable AI and how enterprises can make it a part of their AI strategy. In this post, we’ll dive into the role that data plays in undergirding an AI model’s decisions – and the ability to explain them.
Explainable AI in a Nutshell
Most AI algorithms operate in what’s called a “black box,” where understanding how and why the inputs and outputs work the way they do is virtually impossible. So that means if someone were to ask how an AI model reached the interpretation it did, we’d be hard pressed to explain it. That may not seem problematic until we start relying on AI for decisions that impact human well-being, such as healthcare choices. If a doctor doesn’t have knowledge of all the data and inputs that an AI model used to reach an insight, they could prescribe unnecessary treatment, or conversely, think a patient is healthy when in fact they need immediate care.
On the other hand, explainable AI operates in a glass box. It allows users to see and understand important elements such as the data that went into training a model and why it was chosen; what features of a model were used to reach an insight; and the individual layers that make up the model and how they led to a certain prediction or analysis.
To offer more guidelines on what constitutes explainable AI, the National Institute of Standards developed four principles:
- An AI system should supply “evidence, support, or reasoning for each output.”
- An explanation should accurately reflect the process the system used to arrive at the output.
- An AI system should provide explanations that its users can understand.
- An AI system should operate only under the conditions it was designed for and not provide output when it lacks sufficient confidence in the result.
The Importance of Data for Explainability
Most AI algorithms use deep learning to identify patterns when faced with mountains of training data. And that training data can make or break explainability. If a user doesn’t know or understand the data that went into an AI model, the odds are slim they’ll be able to explain its conclusions. Understanding the underlying training data and process requires the ability to map to the exact dataset that was used to train the model, with the ability to examine that data closely, even years after the model was trained.
That means that one of the best ways to enable or enhance explainability is to keep an eagle eye on the data used to train the model. So, during design, teams must carefully determine where training data will come from, whether it was obtained legally and ethically, whether the data contains bias, and what methods can be used to counteract that bias. Considering that 67% of companies draw from more than 20 data sources, this can be a Herculean task.
It’s also important to intentionally exclude data that is irrelevant to the outcome. For example, a loan approval algorithm should not make decisions based on an irrelevant factor such as zip code (which could lead to biased outcomes), which means that the design team will need to implement measures to ensure that piece of data is not included in the training set or input data.
Since data plays such an important role in explainability, enterprises must develop a strategic approach to its management and governance to ensure that data is available, consistent, legally and ethically sourced, usable, and secure. It’s also important to keep up with best practices for data governance such as complete traceability for the datasets used to train each AI model.
Work with a Knowledgeable Partner
The leaps and bounds of innovation and ability of AI has truly created a whole new frontier for enterprises. As AI evolves and continues to significantly impact human lives, explainability will become the standard so it’s important that your organization understands it – and most importantly – how to implement it. Understanding data’s critical role in AI is a great first step, and can help ensure your AI models are trusted, safe, and secure.
And if you’re looking for help on your AI journey, consider working with the OG of AI. With a focus on AI and ML, we deliver transformational results for our clients by leveraging the latest technology and empowering companies to disrupt, transform, accelerate, and scale. Find out more about us on our website.
