Explainable AI: A Review of Machine Learning Interpretability Methods

more explainable a model, the deeper the understanding that humans achieve in terms of

the internal procedures that take place while the model is training or making decisions.

An interpretable model does not necessarily translate to one that humans are able to

understand the internal logic of or its underlying processes. Therefore, regarding machine

learning systems, interpretability does not axiomatically entail explainability, or vice versa.

As a result, Gilpin et al. [16] supported that interpretability alone is insufficient and that

the presence of explainability is also of fundamental importance. Mostly aligned with the

work of Doshi-Velez and Kim [15], this study considers interpretability to be a broader

term than explainability.

2.2. Evaluation of Machine Learning Interpretability

Doshi-Velez and Kim [15] proposed the following classification of evaluation methods

for interpretability: application-grounded, human-grounded, and functionally-grounded,

subsequently discussing the potential trade-offs among them. Application-grounded eval-

uation concerns itself with how the results of the interpretation process affect the human,

domain expert, end-user in terms of a specific and well-defined task or application. Con-

crete examples under this type of evaluation include whether an interpretability method

results in better identification of errors or less discrimination. Human-grounded evaluation

is similar to application-grounded evaluation; however, there are two main differences:

first, the tester in this case does not have be a domain expert, but can be any human end-

user and secondly, the end goal is not to evaluate a produced interpretation with respect

to its fitness for a specific application, but rather to test the quality of produced interpre-

tation in a more general setting and measure how well the general notions are captured.

An example of measuring how well an interpretation captures the abstract notion of an

input would be for humans to be presented with different interpretations of the input, and

them selecting the one that they believe best encapsulates the essence of it. Functionally-

grounded evaluation does not require any experiments that involve humans, but instead

uses formal, well-defined mathematical definitions of interpretability to evaluate quality

of an interpretability method. This type of evaluation usually follows the other two types

of evaluation: once a class of models has already passed some interpretability criteria via

human-grounded or application-grounded experiments, then mathematical definitions can

be used to further rank the quality of the interpretability models. Functionally-grounded

evaluation is also appropriate when experiments that involve humans cannot be applied

for some reason (e.g ethical considerations) or when the proposed method has not reached

a mature enough stage to be evaluated by human users. That said, determining the right

measurement criteria and metric for each case is challenging and remains an open problem.

2.3. Related Work

The concepts of interpretability and explainability are hard to rigorously define;

however, multiple attempts have been made towards that goal, the most emblematic works

being [14,15].

The work of Gilpin et al. [16] constitutes another attempt to define the key concepts

around interpretability in machine learning. The authors, while focusing mostly on deep

learning, also proposed a taxonomy, by which the interpretability methods for neural

networks could be classified into three different categories. The first one encompasses

methods that emulate the processing of data in order to create insights for the connections

between inputs and outputs of the model. The second category contains approaches

that try to explain the representation of data inside a network, while the last category

consists of transparent networks that explain themselves. Lastly, the author recognises the

promising nature of the progress achieved in the field of explaining deep neural networks,

Entropy 2021, 23, 18

4 of 45

but also highlights the lack of combinatorial approaches, which would attempt to merge

different techniques of explanation, claiming that such types of methods would result in

better explanations.

Adadi and Berrada [17] conducted an extensive literature review, collecting and

analysing 381 different scientific papers between 2004 and 2018. They arranged all of

the scientific work in the field of explainable AI along four main axes and stressed the

need for more formalism to be introduced in the field of XAI and for more interaction

between humans and machines. After highlighting the trend of the community to explore

explainability only in terms of modelling, they proposed embracing explainability in other

aspects of machine learning. Finally, they suggested a potential research direction that

would be towards the composition of existing explainability methods.

Another survey that attempted to categorise the existing explainability methods is

this of Guidotti et al. [19]. Firstly, the authors identified four categories for each method

based on the type of problem that they were created to tackle. One category for explaining

black-box models, one for inspecting them, one for explaining their outcomes, and, finally,

one for creating transparent black box models. Subsequently, they proposed a taxonomy

that takes into account the type of underlying explanation model (explanator), the type of

data used as input, the problem the method encounters, as well as the black box model

that was “opened”. As with works previously discussed, the lack of formality and need

for a definition of metrics for evaluating the performance of interpretability methods was

highlighted once again, while the incapacity of most black-box explainability methods

to interpret models that make decisions based on unknown or latent features was also

raised. Lastly, the lack of interpretability techniques in the field of recommender systems

is identified and an approach according to which models could be learned directly from

explanations is proposed.

Upon identifying the lack of formality and ways to measure the performance of inter-

pretability methods, Murdoch et al. [20] published a survey in 2019, in which they created

an interpretability framework in the hope that it would help to bridge the aforementioned

gap in the field. The Predictive, Descriptive, Relevant (PDR) framework introduced three

types of metrics for rating the interpretability methods, predictive accuracy, descriptive

accuracy, and relevancy. To conclude, they dealt with transparent models and post-hoc

interpretation, as they believed that post-hoc interpretability could be used to elevate the

predictive accuracy of a model and that transparent models could increase their use cases

by increasing predictive accuracy—making clear, that, in some cases, the combination of

the two methods is ideal.

A more recent study carried out by Arrieta et al. [21] introduced a different type

of arrangement that initially distinguishes transparent and post-hoc methods and subse-

quently created sub-categories. An alternative taxonomy specifically for the deep learning

interpretability methods, due to their high volume, was developed. Under this taxonomy,

four categories were proposed: one for providing explanations regarding deep network

processing, one in relation to the explanation of deep network representation, one con-

cerned with the explanation of producing systems, and one encompassing hybrids of

transparent and black-box methods. Finally, the authors dived into the concept of Respon-

sible Artificial Intelligence, a methodology introducing a series of criteria for implementing

AI in organizations.

3. Different Scopes of Machine Learning Interpretability: A Taxonomy of Methods

Different view-points exist when it comes to looking at the the emerging landscape of

interpretability methods, such as the type of data these methods deal with or whether they

refer to global or local properties. The classification of machine learning interpretability

techniques should not be one-sided. There are exist different points of view, which distin-

guish and could further divide these methods. Hence, in order for a practitioner to identify

the ideal method for the specific criteria of each problem encountered, all aspects of each

method should be taken into consideration.

Entropy 2021, 23, 18

5 of 45

A especally important separation of interpretability methods could happen based on

the type of algorithms that could be applied. If their application is only restricted to a

specific family of algorithms, then these methods are called model-specific. In contrast,

the methods that could be applied in every possible algorithm are called model agnostic.

Additionally, one crucial aspect of dividing the interpretability methods is based on the

scale of interpretation. If the method provides an explanation only for a specific instance,

then it is a local one and, if the method explains the whole model, then it is global. At

last, one crucial factor that should be taken into consideration is the type of data on which

these methods could be applied. The most common types of data are tabular and images,

but there are also some methods for text data. Figure 2 presents a summarized mind-

map, which visualizes the different aspects by which an interpretability method could be

classified. These aspects should always be taken into consideration by practitioners, in

order for the ideal method with respect to their needs to be identified.

Figure 2. Taxonomy mind-map of Machine Learning Interpretability Techniques.

This taxonomy focuses on the purpose that these methods were created to serve

and the ways through which they accomplish this purpose. As a result, according to

the presented taxonomy, four major categories for interpretability methods are identified:

methods for explaining complex black-box models, methods for creating white-box models,

methods that promote fairness and restrict the existence of discrimination, and, lastly,

methods for analysing the sensitivity of model predictions.

3.1. Interpretability Methods to Explain Black-Box Models

This first category encompasses methods that are concerned with black-box pre-

trained machine learning models. More specifically, such methods do not try to create

interpretable models, but, instead, try to interpret already trained, often complex models,

Entropy 2021, 23, 18

6 of 45

such as deep neural networks. That is also why they sometimes are referred to as post-hoc

interpretability methods in the related scientific literature.

Under this taxonomy, this category, due to the volume of scientific work around

deep learning related interpretability methodologies, is split into two sub-categories, one

specifically for deep learning methods and one concerning all other black-box models.

For each of these sub-categories, a summary of the included methods is shown in Tables 1

and 2 respectively.

3.1.1. Interpretability Methods to Explain Deep Learning Models

The widespread adoption of deep learning methods, combined with the fact that

it is in their very nature to produce black-box machine learning systems, has led to a

considerable amount of experiments and scientific work around them and, therefore, tools

regarding their interpretability. A substantial portion of attention regarding python tools

is focused on deep learning for images more specifically on the concept of saliency in

images, as initially proposed in [22]. Saliency refers to unique features, such as pixels or

resolution of the image in the context of visual processing. These unique features depict the

visually alluring locations in an image and a saliency map is a topographical representation

of them.

Gradients: first proposed in [23], the gradients explanation technique, as its name

suggests, is gradient-based attribution method, according to which each gradient quantifies

how much a change in each input dimension would a change the predictions in a small

neighborhood around the input. Consequently, the method computes an image-specific

class saliency map corresponding to the gradient of an output neuron with respect to

the input, highlighting the areas of the given image, discriminative with respect to the

given class. An improvement over the initial method was proposed in [24], where the

well-known Krizhevsky network [25] was utilised in order to outperform state-of-the-art

saliency models by a large margin, increasing the amount of explained information by 67%

when compared to state-of-the art. Furthermore, in [26], a task-specific pre-training scheme

was designed in order to make the multi-context modeling suited for saliency detection.

Integrated Gradients [27] is gradient-based attribution a method that attempts to

explain predictions that are made by deep neural network by attributing them to the

network’s input features. It is essentially is a variation on calculating the gradient of the

prediction output with respect to the features of the input, as implemented by the simpler

Gradients method. Under this variation, a much desired property, which is known as

completeness or Efficiency [28] or Summation to Delta [29], is satisfied: the attributions

sum up to the target output minus the target output that was evaluated at the baseline.

Moreover, two fundamental axioms that attribution methods ought to satisfy are identified:

sensitivity and implementation invariance. Upon highlighting that most known attribution

methods do not satisfy these axioms, they propose the integrated gradients method as

a simple way obtain great interpretability results. Another work, closely related to the

integrated gradients method, was proposed in [30], where attributions are used in order to

help identify weaknesses of three question-answer models better than the conventional

models, while also to provide workflow superiority.

DeepLIFT [29] is a popular algorithm that was designed to be applied on top of deep

neural network predictions. The method, as described in [29], is an improvement over its

first form [29], also known as the “Gradient ∗ Input” method, where it was observed that

saliency maps that were obtained using the gradient method can be greatly enhanced by

multiplying the gradient with the input signal—an operation that is essentially a first-order

Taylor approximation of how the output would change if the input were set to zero. The

method’s superiority was demonstrated by showing considerable benefits over gradient-

based methods when applied to models that were trained on natural images and genomics

data. By observing the activation of each neuron, it assigns them contribution scores,

calculated by comparing the difference of the output from some reference output to the

differences of the inputs from their reference inputs. By optionally giving separate consid-

Entropy 2021, 23, 18

7 of 45

eration to positive and negative contributions, DeepLIFT can also reveal dependencies that

are missed by other approaches, such as the Integrated Gradients approach [27].

Guided BackPropagation [31], which is also known as guided saliency, is a variant

of the deconvolution approach [32] for visualizing features learned by CNNs, which can

also be applied to a broad range of network structures. Under this approach, the use

of max-pooling in convolutional neural networks for small images is questioned and

the replacement of max-pooling layers by a convolutional layer with increased stride is

proposed, resulting in no loss of accuracy on several image recognition benchmarks.

Deconvolution, as proposed in [32], is a technique for visualizing Convolutional Neu-

ral Networks (CNNs or ConvNets) by utilising De-convolutional Networks (DeconvNets

or DCNNs), as initially proposed in [33]. DeconvNets use the same components, such as

filtering and pooling, but in reverse fashion: instead of mapping pixels to features, they

apply the opposite. Originally, in [33] DeconvNets were proposed as a way of performing

unsupervised learning; however, in [32] they are not used in any learning capacity, but

rather as a tool to provide insight into the function of intermediate feature layers and pieces

of information of an already trained CNN. More specifically, a novel way of mapping

feature activity in intermediate layers back to the input feature space (pixels in the case of

images) was proposed, showing what input pattern originally caused a given activation in

the feature maps. This is done through a DeconvNet being attached to each of CNN layers,

providing a continuous path back to image pixels.

Class Activation Maps, or CAMs, first introduced in [34], is another deep learning

intrepretability method used for CNNs. More specifically, it’s used to indicate the dis-

criminative regions of an image used by a CNN to identify the category of the image. A

feature vector is created by computing and concatenating the averages of the activations of

convolutional feature maps that are located just before the final output layer. Subsequently,

a weighted sum of this vector is fed to the final softmax loss layer. Using this simple

architecture, the importance of the image regions, pertaining to their classification, can,

therefore, be identified by projecting back the weights of the output layer on to the convo-

lutional feature maps. CAM has two distinct drawbacks: Firstly, in order to be applied, it

requires that neural networks have a very specific structure in their final layers and, for all

other networks, the structure needs to be changed and the network needs to be re-trained

under the new architecture. Secondly, the method, being constrained to only visualising

the final convolutional layers of a CNN, is only useful when it comes to interpreting the

very last stages of the network’s image classification and it is unable to provide any insight

into the previous stages.

Grad-CAM [35] is a strict generalization of CAM that can produce visual explanations

for any CNN, regardless of its architecture, thus overcoming one of the limitations of

CAM. As a gradient-based method, Grad-CAM uses the class-specific gradient information

flowing into the final convolutional layer of a CNN in order to produce a coarse localiza-

tion map of the important regions in the image when it comes to classification, making

CNN-based models more transparent. The authors of Grad-CAM also demonstrated how

the technique can be combined with existing pixel-space visualizations to create a high-

resolution class-discriminative visualization, Guided Grad-CAM. By generating visual

explanations in order to better understand image classification of popular networks while

using both Grad-CAM and Guided Grad-CAM, it was shown that the proposed techniques

outperform pixel-space gradient visualizations (Guided Backpropagation and Deconvo-

lution) when evaluated in terms of localisation (the ability to localise objects in images

using holistic image class labels only) and faithfulness (the ability to accurately explain

the function learned by a model). While an improvement over CAM, Grad-CAM has its

own limitations, the most notable including its inability to localize multiple occurrences

of an object in an image, due its partial derivative assumptions, its inability to accurately

determine class-regions coverage in an image, and the possible loss in signal due the

continual upsampling and downsampling processes.

Entropy 2021, 23, 18

8 of 45

Grad-CAM++ [36] is an extension of the Grad-CAM method that provides better

visual explanations of CNN model predictions. More specifically, object localization is

extended to multiple object instances in a single image while using a weighted combination

of the positive partial derivatives of the last convolutional layer feature maps with respect

to a specific class score as weights to generate a visual explanation for the corresponding

class label. This is especially helpful in multi-label classification problems, while the

different weight assigned to each pixel makes it possible to capture the importance of each

pixel separately in the gradient feature map.

Layer-wise Relevance Propagation (LRP) [37] is a “decomposition of nonlinear classi-

fiers” technique that brings interpretability to highly complex deep neural networks by

propagating their predictions backwards. The proposed propagation procedure satisfies

a conservation property, whereby the magnitude of any output is remains intact, as it is

backpropagated through the lower-level layers of the network: Starting from the output

neurons going all the way back to the input-layer neurons, each neuron redistributes to

the lower layer the same amount of information as it received from the higher layer. The

method can be applied to various data types, such as images, text, and more, as well as

various neural network architectures.

By pointing out and exploiting the fact that the gradient of the loss function with

respect to the input can be interpreted as a sensitivity map, Smilkov et al. [38] created

SmoothGrad, a method that can be applied in order to reduce noise in order visually

sharpen such sensitivity maps. SmoothGrad can be combined with other sensitivity map

algorithms, such as the Integrated Gradients [27] and Guided BackPropagation [31], in

order to produce enhanced sensitivity maps—more specifically, two smoothing approaches

were explored and experimented with: The first one, which had an excellent smoothing

impact, calculates the average of maps made from many small perturbations of a given

instance, while the second perturbs the data with random noise and then performs the

training step. The experiments showed that these two techniques can have an additive

effect, and combining them provides superior results to applying them separately. Upon

performing a series of experiments, the authors conclude that the estimated smoothed gra-

dient leads to sharper visualisations and more coherent sensitivity maps when compared

to the non-smoothed gradient.

In order to interpret the predictions of deep neural networks for images, the RISE

algorithm [39] creates a saliency map for any black-box model, indicating how important

each pixel of the image with respect to the network’s prediction. The method follows a

simple yet powerful approach: each input image is multiplied element-wise with random

masks and the resulting image is subsequently fed to the model for classification. The

model produces a probability-like score for the masked images with respect to each of the

available classes and a saliency map for the original image is created as a linear combination

of the masks. The coefficients of this linear combination are calculated while using the

score that was produced by the model for the corresponding masked inputs with respect

to target class.

Entropy 2021, 23, 18

9 of 45

Table 1. Interpretability Methods to Explain Deep Learning Models.

Tool

Category

Local vs.

Global

Model Specific vs.

Model Agnostic

Data Type

Citations/

Year

Year

DeepExplain

iNNvestigate

tf-explain

Grad-CAM

tf-explain

CAM

iNNvestigate

DeepExplain

iNNvestigate

tf-explain

DeepExplain

iNNvestigate

Integrated Gradients

tf-explain

alibi

Skater

Deep Visualization

Toolbox

DeepExplain

iNNvestigate

The LRP Toolbox

Skater

DeepExplain

DeepLift

iNNvestigate

tf-explain

Skater

iNNvestigate

iNNvestigate

tf-explain

tcav

rationale

Grad-CAM++

RISE

iNNvestigate

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

PH

L

L

L

L

L

L

L

L

L

L

L

L

L

L

L

L

Specific

img

1548.3

2014

Specific

Specific

Specific

Specific

Specific

img

img

img

img

img

txt

tab

797.8

607.8

365.3

2017

2016

2014

278.3

2013

247

2017

Specific

img

221.7

2015

Specific

img

txt

217.8

2015

Specific

img

211.5

2017

Specific

Specific

Specific

Specific

Specific

Specific

Specific

img

img

img

txt

img

img

img

131.5

113.3

95

81.4

81

43.3

41.8

2017

2017

2018

2016

2018

2018

2017

Ref

[32]

[35]

[34]

[31]

[23]

[27]

[40]

[37]

[29]

[41]

[38]

[42]

[43]

[36]

[39]

[44]

In [42], the idea of Concept Activation Vectors (CAVs) was introduced, providing

a human-friendly interpretation of a neural network internal state; an intuition of how

sensitive a prediction is to a user-defined concept and how important the concept is to the

classification itself. One of the issues with saliency maps is that concepts in an image, such

as the “human” concept or the “animal” concept, cannot be expressed as pixels and are not

in the input features either and therefore cannot be captured by saliency maps. To address

this CAVs try to provide a translation between the input vector space and the high-level

concept space; a CAV corresponding to a concept is essentially a vector in the direction

of the values (the result of activation functions in a network’s neurons) of that concept’s

set of examples. By utilising CAVs, the TCAV method provides a quantitative measure

Entropy 2021, 23, 18

10 of 45

of importance of a concept if and only if the network has learned about it. Furthermore,

TCAV can reveal any concept learnt, even if it was not explicitly tagged within the training

set or even if was not part of the input feature set.

Yosinski et al. [40] proposed applying regularisation as an additional processing

step in the saliency map creating process. More specifically, by introducing four primary

regularization techniques, they enforced stronger prior distributions in order to promote

bias towards more recognisable and interpretable visualisations. They showed that the

best results were obtained when the different regularisers were combined, while each of

these regularisation methods can also individually enhance interpretability.

In [43], an interpretability technique for neural networks operating in the natural

language processing (NLP) domain was proposed. Under this approach, smaller, tailored

pieces of the original input text are extracted and then used as input in order to try and

produce the same output prediction as the original full-text input. These small pieces,

called rationales, provide the necessary explanation and justification for the output in terms

of the input. The architecture consists of two components, a generator and an encoder,

which are trained to function well as a whole. The generator produces candidate rationales,

and the encoder uses them to produce predicted probability scores. The generator and the

encoder are trained jointly, and, through the minimization of the cost function, it is decided

which candidates will be characterised as rationals. Essentially, the two components work

together in order to find subsets of text that are highly associated with the predicted score.

Deep Taylor decomposition [41] is a method that decomposes a neural network’s

output, for given input instance, into contributions of this instance by backpropagating the

explanations from the output layer to the input. Its usefulness was demonstrated within

the computer vision paradigm, in order to measure the importance of single pixels in image

classification tasks; however, the method can also be applied to different types of data

as both a visualization tool as well as a tool for more complex analysis. The proposed

approach has strong links to relevance propagation; the theoretical connections between

the Taylor decomposition of a function and rule-based relevance propagation techniques

are thoroughly discussed, demonstrating a clo...