Explainable artificial intelligence (XAI)

Explainable AI (XAI) refers to the set of technologies and practices required to make the outputs of an AI model understandable and interpretable by human experts.

Explainable AI (XAI) refers to the set of technologies and practices required to make the outputs of an AI model understandable and interpretable by human experts.

via DARPA

The "black box AI" problem

Artificial Intelligence is sometimes characterized as a black box system because it can be difficult to determine how a particular output or behavior resulted from a given set of inputs.

This presents a number of challenges for AI researchers and engineers in academia and in the private sector:

  • As machine learning and AI techniques become part of the standard analytical toolkit across many scientific disciplines, unexplainable "black box" machine learning models make reproducing scientific experiments and analysis difficult.
  • In commercial settings, unexplainable AI models may produce results which are difficult to audit, and thus difficult to trust. Black box AI models may also be biased in ways that are difficult to detect and resolve.

Risks imposed by mysterious, unexplainable behavior in AI models prompted AI and machine learning researchers to seek out ways to produce more explainable models and build artificial intelligence systems which can be understood and appropriately trusted.

Opening the black box

There are several approaches to the challenge of "opening the black box," including:

  • Model explanation, wherein the explanation involves the global logic of the black box classifier
  • Outcome explanation, wherein the goal is to locally understand the reasons for the decision of a particular output or result
  • Model inspection, where the objective is to understand how the black box behaves internally, which can involve using alternate inputs and other tools to unpack the inner workings of a model

Research programs

There are many research efforts into explainable AI by academic institutions, the private sector, and governments. DARPA—the research and development arm of the U.S. military—launched its Explainable Artificial Intelligence program in August 2016.

Companies leveraging Explainable AI

Timeline

August 11, 2016

DARPA launches its Explainable Artificial Intelligence research program

In its announcement, DARPA said that "The goal of Explainable Artificial Intelligence (XAI) is to create a suite of new or modified machine learning techniques that produce explainable models that, when combined with effective explanation techniques, enable end users to understand, appropriately trust, and effectively manage the emerging generation of Artificial Intelligence (AI) systems."

People

Name
Role
LinkedIn

Further reading

Title
Author
Link
Type
Date

A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI

Erico Tjoa, Cuntai Guan

August 2015

Artificial Intelligence Confronts a 'Reproducibility' Crisis

Gregory Barber

Web

September 16, 2019

Explainable AI in industry: practical challenges and lessons learned: implications tutorial

Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly

Conference paper

January 2020

Explainable Artificial Intelligence

Dr. Matt Tureck

Web

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera

October 22, 2019

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI

Shane T. Mueller, Robert R. Hoffman, William Clancey, Abigail Emrey, Gary Klein

February 5, 2019

Interesting resources related to XAI (Explainable Artificial Intelligence)

Przemysław Biecek

Github repository

Towards Explainable Artificial Intelligence

Wojciech Samek, Klaus-Robert Müller

September 26, 2019

Documentaries, videos and podcasts

Title
Date
Link

Explainable AI for Science and Medicine

Explaining the Predictions of Machine Learning Models with Carlos Guestrin - The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

October 9, 2016

Real World Model Explainability with Rayid Ghani - Talk #283

July 18, 2019

The Problem with Black Boxes with Cynthia Rudin - Talk #290

August 12, 2019

Companies

Company
CEO
Location
Products/Services

References

Golden logo
Text is available under the Creative Commons Attribution-ShareAlike 4.0; additional terms apply. By using this site, you agree to our Terms & Conditions.