Explainable artificial intelligence (XAI)

Explainable AI (XAI) refers to the set of technologies and practices required to make the outputs of an AI model understandable and interpretable by human experts.

via DARPA

The "black box AI" problem

Artificial Intelligence is sometimes characterized as a black box system because it can be difficult to determine how a particular output or behavior resulted from a given set of inputs.

This presents a number of challenges for AI researchers and engineers in academia and in the private sector:

• As machine learning and AI techniques become part of the standard analytical toolkit across many scientific disciplines, unexplainable "black box" machine learning models make reproducing scientific experiments and analysis difficult.
• In commercial settings, unexplainable AI models may produce results which are difficult to audit, and thus difficult to trust. Black box AI models may also be biased in ways that are difficult to detect and resolve.

Risks imposed by mysterious, unexplainable behavior in AI models prompted AI and machine learning researchers to seek out ways to produce more explainable models and build artificial intelligence systems which can be understood and appropriately trusted.

Opening the black box

There are several approaches to the challenge of "opening the black box," including:

• Model explanation, wherein the explanation involves the global logic of the black box classifier
• Outcome explanation, wherein the goal is to locally understand the reasons for the decision of a particular output or result
• Model inspection, where the objective is to understand how the black box behaves internally, which can involve using alternate inputs and other tools to unpack the inner workings of a model

Research programs

There are many research efforts into explainable AI by academic institutions, the private sector, and governments. DARPA—the research and development arm of the U.S. military—launched its Explainable Artificial Intelligence program in August 2016.