Explainable AI (XAI) refers to the set of technologies and practices required to make the outputs of an AI model understandable and interpretable by human experts.

via DARPA
Artificial Intelligence is sometimes characterized as a black box system because it can be difficult to determine how a particular output or behavior resulted from a given set of inputs.
This presents a number of challenges for AI researchers and engineers in academia and in the private sector:
- As machine learning and AI techniques become part of the standard analytical toolkit across many scientific disciplines, unexplainable "black box" machine learning models make reproducing scientific experiments and analysis difficult.
- In commercial settings, unexplainable AI models may produce results which are difficult to audit, and thus difficult to trust. Black box AI models may also be biased in ways that are difficult to detect and resolve.
Risks imposed by mysterious, unexplainable behavior in AI models prompted AI and machine learning researchers to seek out ways to produce more explainable models and build artificial intelligence systems which can be understood and appropriately trusted.
There are several approaches to the challenge of "opening the black box," including:
- Model explanation, wherein the explanation involves the global logic of the black box classifier
- Outcome explanation, wherein the goal is to locally understand the reasons for the decision of a particular output or result
- Model inspection, where the objective is to understand how the black box behaves internally, which can involve using alternate inputs and other tools to unpack the inner workings of a model
There are many research efforts into explainable AI by academic institutions, the private sector, and governments. DARPA—the research and development arm of the U.S. military—launched its Explainable Artificial Intelligence program in August 2016.
Companies leveraging Explainable AI
Timeline
In its announcement, DARPA said that "The goal of Explainable Artificial Intelligence (XAI) is to create a suite of new or modified machine learning techniques that produce explainable models that, when combined with effective explanation techniques, enable end users to understand, appropriately trust, and effectively manage the emerging generation of Artificial Intelligence (AI) systems."
People
Further reading
A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
Erico Tjoa, Cuntai Guan
August 2015
Artificial Intelligence Confronts a 'Reproducibility' Crisis
Gregory Barber
Web
September 16, 2019
Explainable AI in industry: practical challenges and lessons learned: implications tutorial
Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly
Conference paper
January 2020
Explainable AI: The Basics
PDF / Policy Brief
November 2019
Explainable Artificial Intelligence
Dr. Matt Tureck
Web
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera
October 22, 2019
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
Shane T. Mueller, Robert R. Hoffman, William Clancey, Abigail Emrey, Gary Klein
February 5, 2019
Interesting resources related to XAI (Explainable Artificial Intelligence)
Przemysław Biecek
Github repository
Towards Explainable Artificial Intelligence
Wojciech Samek, Klaus-Robert Müller
September 26, 2019
Documentaries, videos and podcasts
Explaining the Predictions of Machine Learning Models with Carlos Guestrin - The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
October 9, 2016
Real World Model Explainability with Rayid Ghani - Talk #283
July 18, 2019
The Problem with Black Boxes with Cynthia Rudin - Talk #290
August 12, 2019