Explainable artificial intelligence (XAI)

Explainable AI (XAI) refers to the set of technologies and practices required to make the outputs of an AI model understandable and interpretable by human experts.

Explainable AI (XAI) refers to the set of technologies and practices required to make the outputs of an AI model understandable and interpretable by human experts.

The "black box AI" problem

Artificial Intelligence is sometimes characterized as a black box system because it can be difficult to determine how a particular output or behavior resulted from a given set of inputs.

This presents a number of challenges for AI researchers and engineers in academia and in the private sector:

  • As machine learning and AI techniques become part of the standard analytical toolkit across many scientific disciplines, unexplainable "black box" machine learning models make reproducing scientific experiments and analysis difficult.
  • In commercial settings, unexplainable AI models may produce results which are difficult to audit, and thus difficult to trust. Black box AI models may also be biased in ways that are difficult to detect and resolve.

Risks imposed by mysterious, unexplainable behavior in AI models prompted AI and machine learning researchers to seek out ways to produce more explainable models and build artificial intelligence systems which can be understood and appropriately trusted.

Opening the black box

There are several approaches to the challenge of "opening the black box," including:

  • Model explanation, wherein the explanation involves the global logic of the black box classifier
  • Outcome explanation, wherein the goal is to locally understand the reasons for the decision of a particular output or result
  • Model inspection, where the objective is to understand how the black box behaves internally, which can involve using alternate inputs and other tools to unpack the inner workings of a model
Research programs

There are many research efforts into explainable AI by academic institutions, the private sector, and governments. DARPA—the research and development arm of the U.S. military—launched its Explainable Artificial Intelligence program in August 2016.

Companies leveraging Explainable AI


August 11, 2016
DARPA launches its Explainable Artificial Intelligence research program

In its announcement, DARPA said that "The goal of Explainable Artificial Intelligence (XAI) is to create a suite of new or modified machine learning techniques that produce explainable models that, when combined with effective explanation techniques, enable end users to understand, appropriately trust, and effectively manage the emerging generation of Artificial Intelligence (AI) systems."



Further reading


A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI

Erico Tjoa, Cuntai Guan

August 2015

Artificial Intelligence Confronts a 'Reproducibility' Crisis

Gregory Barber


September 16, 2019

Explainable AI in industry: practical challenges and lessons learned: implications tutorial

Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly

Conference paper

January 2020

Explainable AI: The Basics

The Royal Society

PDF / Policy Brief

November 2019

Explainable Artificial Intelligence

Dr. Matt Tureck


Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI

Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera

October 22, 2019

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI

Shane T. Mueller, Robert R. Hoffman, William Clancey, Abigail Emrey, Gary Klein

February 5, 2019

Interesting resources related to XAI (Explainable Artificial Intelligence)

Przemysław Biecek

Github repository

Towards Explainable Artificial Intelligence

Wojciech Samek, Klaus-Robert Müller

September 26, 2019

Documentaries, videos and podcasts


Explainable AI for Science and Medicine

Explaining the Predictions of Machine Learning Models with Carlos Guestrin - The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

October 9, 2016

Real World Model Explainability with Rayid Ghani - Talk #283

July 18, 2019

The Problem with Black Boxes with Cynthia Rudin - Talk #290

August 12, 2019




Golden logo
Text is available under the Creative Commons Attribution-ShareAlike 4.0; additional terms apply. By using this site, you agree to our Terms & Conditions.