the perceived difference.

The algorithm or method used to generate the adversarial

example is often referred to as an adversarial attack. A

targeted attack generates an adversarial example that gets

classified as a specific, attacker defined, target class, whereas

a non targeted attack merely causes a misclassification to any

incorrect target class. Many adversarial example generation

methods have been invented in recent years. Some of the most

notable, which are among the ones we use for our evaluation

include:

• Fast Gradient Sign Method (FGSM) [19] - a basic tech-

nique that

involves taking a single step in the input

space in the direction of the gradient of the model’s

cost function with magnitude equal to the max allowed

perturbation norm ((cid:15)).

• Basic Iterative Method (BIM) and its variation Projected

Gradient Descent (PGD) [5], [6] - a natural extension of

FGSM that takes multiple FGSM-like steps with smaller

step sizes adding up to less than the maximum allowed

perturbation size ((cid:15)).

• Carlini & Wagner (C&W) [4] - an attack that formulates

the problem of finding an adversarial examples as an

optimization problem with a cleverly chosen loss function

tailored for each metric.

Adversarial attacks can be further divided to white box and

black box. In the white box scenario, an attacker has full access

to the attacked classification model, including its internal struc-

ture and parameters/weights. In black box attacks, the attacker

can only feed the model with inputs and observe the outputs

but doesn’t have access to its internal state. An important

property of many adversarial attacks is transferability [29].

According to this property adversarial examples generated

against one model can fool other models as well. This allows

converting a white-box into a black-box attack by training a

surrogate model and generating an adversarial example against

it that can be successfully fool the original model.

B. Adversarial Defenses

Previously proposed defense methods against adversarial

attacks can be categorized as methods that aim at improving

the robustness of the trained model

to adversarial attacks

and methods that aim at detecting adversarial examples.

Methods for generating robust models include adversarial

training [19], Defensive Distillation [7], Gradient Obfuscation,

Feature Squeezing [13] and more.

In this research we propose a method for detecting adversar-

ial examples. Most previously proposed methods for detecting

adversarial examples attempted to identify irregularities in

the input data, or in the internal behavior (e.g.,

internal

layer activations) of the model, while others take a more

active approach that involves transforming the inputs [13] or

modifying the training process [16] to improve the detector

performance.

C. Understanding adversarial examples

Ilyas et al.

[18] argue that the existence of adversarial

examples is actually an intrinsic property of the dataset itself.

They introduce that notion of robust and non-robust features.

Non-robust features are highly predictive, yet very fragile and

prone to change drastically due to small perturbations of the

input. Robust features, on the other hand, are features that

are both highly predictive and do not change easily by small

change in the input. One can think of robust features as

features that capture some important, high-level, feature of

the target class - such as the presence of wheels and windows

for cars, whereas non-robust features are seemingly random

patterns that aren’t noticeable by human beings, but emerge as

highly predictive during the training process. Ilyas et al. show

that the existence of adversarial examples, as well as their

transferability across different classification models, naturally

arises from the existence of non-robust features since they

allow small perturbations in the input to cause major changes

in value of these highly-predictive features.

D. Explainable AI

Explainable AI (XAI) is an emerging researched field in

machine learning with the purpose of allowing users to un-

derstand, trust, and effectively manage the next generation of

AI solutions [21]. Most of the XAI methods developed in

recent years are meant to explain supervised machine learning

models. For example, the LIME [30] method introduced for

explaining the prediction using a local model; the DeepLIFT

method [31], which uses back propagation through all of

the neurons in the network to explain the output; and the

SHAP [25] method, which is a unified approach that aims

to explain the model output using shapely values - a concept

borrowed from game theory where it is used to calculate the

relative contributions of different players in a coalition. In the

context of XAI, they are used for estimating the contribution

of a specific input or neuron to a model’s decision. The need to

explain the output is especially important in anomaly detection

based on deep learning models, because usually in this case

not all of the anomaly types are known (labeled). In this

research we use the SHAP DeepExplainer method, which is a

variation of the SHAP algorithm that is specifically optimized

for explaining DNN models.

III. RELATED WORK

Previous works suggested methods for detecting adversarial

examples. A summary of these works is presented in Table I.

The table presents a succinct summary of the detection concept

and evaluation setup and results of each detector. In addition,

we summarize the most important pros and cons of each

detector. In this research we propose a novel approach for

detecting adversarial sample which was not proposed before.

In addition, we conduct a more comprehensive evaluation by

checking the models ability for cross attack generalization as

well as evaluating on a larger quantity of diverse attack types,

thus providing a higher confidence in our model’s ability to

adapt to real-world challenges.

IV. ROBUSTNESS THROUGH EXPLAINABILITY

Adversarial evasion attacks change the values of non-robust

features, while largely leaving robust features intact [18].

This is because applying an effective modification to robust

features requires significant changes to the input. Conse-

quently, we hypothesize that we should see different patterns

in the importance of robust vs. non-robust features in the

classification of normal and adversarial inputs, with the lat-

ter relying more heavily on non-robust features. We try to

leverage this hypothesized property of adversarial examples by

utilizing explainable AI methods (XAI) for interpreting model

predictions.

Consequently, for each input to be classified as adversarial

or normal, we utilize SHAP [25] to compute the importance

scores of the neurons of the penultimate layer of the clas-

sification model. Then, we use these importance scores as

features for our adversarial example detector. The reason for

interpreting the penultimate layer (instead of the input layer

for instance) is because the neurons of this layer actually form

high-level features of the original classification model [32].

Figure 1 provides an illustration that supports our hypoth-

esis. On the left side of the figure, we can see three normal

examples of the same class “cat”, and on the right side we

can see three normal examples of another class “automobile”.

In the middle of the figure we can see a normal (original)

example from the class “automobile” and a perturbation of

that example after applying a targeted (target class “cat”)

PGD L2 attack [6]. Below each image (example) we present

the SHAP XAI signature of the image, such that each pixel

in the signature at coordinates (row = i,col = j) contains

the SHAP value of neuron i for target class j. Red pixels

denote positive contributions of their respective neurons for

steering the model’s decision towards the respective target

class, whereas blue pixels denote a negative contribution,

steering the model away. The intensity of the color denotes

the magnitude of the positive or negative contribution, with

white/transparent pixels denoting no contribution at all. From

a birds eye view of the figure it can be observed that the

XAI signatures of images of the same class (automobiles or

cats) are similar, while different classes have different XAI

signatures. In can be also observed that although the original

and perturbed automobile examples look the same, their XAI

signatures are different. A closer look, however, uncovers

even more intriguing properties: The normal car contains five

relatively distinctive rows in their XAI signatures (two near

the top, two near the bottom and one closer to the middle).

Moreover, the bright red pixels in these rows are located

in columns 1 and 9 which correspond to the target classes

“automobile” and ”truck”. On the left hand side of the figure,

the three normal cat examples share a similarly looking lump

of red pixels in the upper middle part of the XAI signature.

Moving over to the adversarial automobile example, we can

see that it shares two of the five distinctive rows with the

normal cars, and a red lump in the upper middle with the cats.

Thus, a mixture of “automobile” and “cat” features plays an

important role in the decision of the underlying classifier for

this adversarial example. Although this is merely a speculation

and further research is required to draw strong conclusions, but

we hypothesize that this behaviour is perfectly aligned with the

notion of robust and non-robust features: The two distinctive

rows that

the adversarial attack failed to alter correspond

to robust features of the “automobile” class, whereas the

remaining three rows that did disappear correspond to non-

robust “automobile” features. Likewise - the part of the red

lump that transferred from the normal cats to the adversarial

one correspond to non-robust “cat” features, while the part

of the lump the didn’t transfer corresponds to robust “cat”

features.

To explore the dataset

in more depth, we trained a

UMAP [33] dimensionality reducer on the train set and used

it to project the test set onto the embedding space of the train

set.

In Figure 2 we can clearly see ten distinct clusters of normal

samples, one for each target class, and one cluster containing

Ref

[10]

[14]

[11]

Binary detector fed by

activation of internal

layers

Cluster activations of

internal layers and classify

adversarial examples

based on movements

between different clusters

Extract features to

construct two

unsupervised detectors and

also an ensemble of them:

1. Density estimated of

feature space of last

hidden layer 2. Bayesian

uncertainty estimates,

available in dropout neural

networks.

[12]

Statistical tests on raw

inputs

Density estimate anomaly

detector like [11];

additionally, change

training loss function of

defended model from

cross entropy to Reverse

Cross Entropy (RCE) to

improve detection rates

Decrease input resolution

(bit depth reduction, local

smoothing), making adv

attacks more difficult and

detection easier by

comparing model output

for original and

transformed inputs

Based on statistical

differences in the

distribution of logits of

f(x+(cid:15))

[15]

[13]

[16]

Concept

Datasets

Attacks

CIFAR-10,

MNIST,

IMAGENET-

10

FGSM, BIM

(L2, L∞),

DeepFool

(L2, L∞)

CIFAR-10,

MNIST

C&W (L2)

Main Results

Accuracy: between 0.79 and

0.97 (average 0.87) when

training and testing on the

same attack; cross attack

scores are much lower

CIFAR10 AUC=0.92 (0.95

for correctly classified only)

; MNIST AUC=0.91

Pros

Evaluating transferability

between different attacks

and perturbation budgets

Good performance against

the strong C&W attack

Cons

Relatively low detector

accuracy, especially in

generalization scenario;

evaluating on a small set

of attacks

Evaluating against the

C&W L2 attack only

(might not perform well

against other attacks)

MNIST,

CIFAR-10,

SVHN

MNIST,

DREBIN ,

MicroRNA

FGSM, BIM,

JSMA, C&W

CIFAR-10: Average AUC of

0.8554

Unsupervised approach (no

need for adv samples for

training)

Low detector performance

(AUC), evaluating on a

weak CIFAR-10 classifier

FGSM, JSMA

(on MNIST)

Detection rate: FGSM 99%

JSMA 80%

MNIST,

CIFAR-10

FGSM, BIM,

ILCM, JSMA,

C&W

AUC: FGSM 99.7, BIM

100.0, ILCM 84.2, JSMA:

85.8, C&W: 95.3; RCE

model: C&W 91.8, ILCM

93.9, JSMA 95.4, C&W

98.2, FGSM 99.7, BIM 100

Generic approach

(applicable to different

domains and model types

such as SVM and Decision

Trees)

Evaluating on weak

attacks; low performance

on adv samples generated

using JSMA

Good performance on RCE

trained model; unsupervised

approach (no need for adv

samples for training)

Requires customizing the

training of the defended

model

CIFAR-10,

MNIST,

IMAGENET

CIFAR-10,

IMAGENET

FGSM, BIM,

C&W (L0,

L2, L∞),

DeepFool,

JSMA

Train on PGD

L∞ and Eval

on PGD L∞,

PGD L2,

C&W L2

CIFAR-10: TPR of 0.845

(FPR=0.05)

Evaluating on a diverse set

of adversarial attacks; good

detection performance

against the strong C&W

attack

Weak performance against

the BIM, DeepFool,

JSMA, FGSMS attacks

(relatively weak overall

detection performance)

PGD L∞: 99.1% TPR /

0.2% FPR, PGDŁ2: 96.1% /

1% C&W L 2: 91.6% /

4.8%

High detection performance;

generalization across

adversarial attacks

Not evaluating on a

diverse set of attacks

TABLE I

SUMMARY OF RELATED WORKS.

adversarial examples. From this separation we deduced that

the computed SHAP values would make good features for

our detector. Figure 3 shows only the adversarial examples,

projected onto the same embedding space as in Figure 2.

The wide spatial dispersion of different types of adversarial

examples all around the clusters of adversarial examples hints

that we could expect our detector to generalize well when

tested on types of adversarial examples it wasn’t trained on.

V. PROPOSED METHOD

The proposed solution consists of three main phases (Fig-

ure 4): creating a repository of normal and adversarial exam-

ples, generating XAI signatures, and detector construction.

A. Notation

• f (·) - a neural network based classifier

• f [i](·) - the output of the ith neural network layer (0 ≤

i ≤ l), where f [0](·) is the input layer and f [l](·) is the

final softmax output.

• x - input vector

• Y (x) - ground truth label of x

B. Creating a repository of normal and adversarial examples

In this phase, a repository of normal and adversarial samples

is generated. The normal examples are randomly sampled from

the dataset used to train f (·). The adversarial examples are

generated by applying a variety of state-of-the-art adversarial

attack algorithms on f (·). When generating adversarial exam-

ples, it is crucial to investigate the attack’s hyperparameters

such as the distance metrics (e.g., L0, L1, L2, L∞), pertur-

bation budget, number of iteration, attack step and etc) [28].

Fuzzing over the various hyperparameters produces different

types of perturbations (i.e., attacks) and consequently increases

the generalization capability of the detection model.

Algorithm 1 outlines the process of generating a represen-

tative set of adversarial examples. As can be seen, in each

iteration of the algorithm we randomly select: a normal sample

from the dataset used to train the classification model (line

12); a combinations of attack method, distance metrics and

attack preferences (lines 13-15); and a target class, which is

different from the ground truth (line 16). Then, for each tuple

(consisting of: the sample, distance metric, attack preferences,

Fig. 1.

Illustrating the XAI signatures of examples from different classes, as well as original and adversarial example.

Algorithm 1 Generating Adversarial Examples

1: Inputs:

2:

3:

4:

5:

6:

Xnormal ← sampled normal examples

L ← set of possible labels

M ← set of attack methods

D ← set of distance metrics

P (m) ← set of preferences for attack method m ∈

M

f (·) ← target classifier

i ← number of samples to generate

7:

8:

9: procedure GENERATEADVERSARIALEXAMPLES(Xnormal

, L, M , D , P (m) ,f (·), i)

Xadversarial ← φ

while i > 0 do

x ← RandomSample(Xnormal)

m ← RandomSample(M )

d ← RandomSample(D)

p ← RandomSample(P (m))

target ← RandomSample(L ∩ Y (x))

x∗ ← m(x, d, p, target, f (·))

if f (x∗) == target then

Xadversarial ← Xadversarial ∪ x∗

end if

i ← i − 1

10:

11:

12:

13:

14:

15:

16:

17:

18:

19:

20:

21:

22:

end while

return Xadversarial

23:

24: end procedure

to interpret the neurons of the penultimate layer f [l−1](·).

The outcome of this application is n SHAP values for each

output in f [l−1](·), where n represent the number of target

classes (i.e., SHAP produces a single value for each output

and class). The XAI signature of a given sample is defined as

the concatenation of all SHAP values into a flat floating-point

vector (i.e., the size of each signature is to n ∗ |f [l−1](·)|).

Fig. 2. UMAP visualization of XAI signatures for normal and adversarial

examples (CIFAR-10)

Fig. 3. UMAP visualization of XAI signatures for different adversarial

examples (CIFAR-10)

target class and classification model), we execute the attack to

generate an adversarial example (line 17). If the attack ends

successfully (i.e, the classification of the perturbation using

the targeted classifier equal to the target class) we store it in

our repository.

C. Generating XAI signatures

In this phase, we utilize SHAP to generate an XAI signature

for each sample in the dataset (both normal and adversar-

ial). Specifically, we apply the SHAP DeepExplainer [25]

adversarial example(target = cat)normal examples(car)original example(car)normal example(cat)XAI signatureXAI signatureXAI signatureXAI signatureadversarialairplaneautomobilebirdcatdeerdogfroghorseshiptruckProjectedGradientDescent-LinfCarliniL2Method-L2SpatialAttack-L2FastGradientMethod-L2FastGradientMethod-LinfLBFGSAttack-L2ProjectedGradientDescent-L2ElasticNet-L1HopSkipJump-L2CarliniLInfMethod-LinfDeepFool-L2HopSkipJump-LinfGaussianBlurAttack-L2Fig. 4. Detector training process.

Normal signatures are used as a baseline for modeling the

behavior of the normal decision-making procedure within the

activation space. Attack signatures are used for modeling

the effect of different perturbations on the decision-making

procedure.

It should be mentioned that in a production implemen-

tation of this approach,

the repository should be updated

continuously with attack signatures generated using newly

discovered attacks. Maintaining an updated repository will

improve the performance of the proposed method in detecting

new attack classes. Nevertheless, a good detector must be able

to generalize known attacks in order to detect unknown attacks

(we discuss this topic in Section VI).

D. Training the detector

In this phase, we train a supervised binary detector to

discriminate between normal and adversarial samples, based

on their XAI signatures. We use the SHAP values from

our generated dataset as the samples’ features to train the

classifier. Any standard supervised model can be used to

train the detector based on these features. In this research we

trained a fully connected feed forward neural network detector.

At inference time, given a sample to classify as normal or

adversarial, we compute the sample’s SHAP values (in the

XAI signature phase) and feed the output into our binary

classifier to classify the sample as adversarial or normal.

VI. EVALUATION

In our evaluation we aimed to answer the following two

research questions:

RQ1: What is the baseline performance of the detector when

the train and test sets contain adversarial examples generated

using the same attacks and parameters?

RQ2: Can the detector generalize to adversarial examples

generated by attacks that were unknown during training?

A. Evaluation setup

We evaluated our proposed detection method using the

following two image classification use cases:

[CIFAR,ResNet56] The CIFAR-10 dataset

[26] with the

ResNet-56 classification model [34]. The model achieves a

93.39% accuracy on the CIFAR-10 test set.

[MNIST,CNN] The MNIST handwritten digits dataset [27]

with a model architecture taekn from the Keras MNIST exam-

ple (https://keras.io/examples/mnist cnn/). The model achieves

a 99.25% accuracy on the MNIST test set.

Adversarial example generation.: We generated adversarial

examples using both the Foolbox [35] and Adversarial Robust-

ness Toolbox [36] frameworks. The SHAP explanations were

generated using the SHAP framework [25].

Table II presents the number or normal and adversarial

samples used for training and testing the detection model (for

both the CIFAR-10 and MNIST datasets).

Dataset Train/Test

Normal

Adversarial

cifar10

cifar10

mnist

mnist

train

test

train

test

19463

9339

27239

9910

TABLE II

DATASETS DESCRIPTION.

10134

7995

26500

9679

We used a variety of attack methods (see Figure 6) for

generating the adversarial examples.

SHAP values computation.: For

the [CIFAR,ResNet]

model, we compute SHAP values on the last (and only) fully

connected layer of the model which has a size of 64 neurons.

This gives a total of 640 features per sample (64 features for

each one of the ten target classes). For the [MNIST,CNN]

model, we compute SHAP values on the last fully connected

layer of the model, which has a size of 128 neurons, and a

total of 1280 features per sample.

Adversarial ExampleGeneratorTarget Classifier𝑓(⋅)SHAPDeep ExplainerAttack AlgorithmnormalsampleImage Knowledge-baseadversarialsampleXAI signatures Knowledge-baseAttack Hyperparametersadversarial/normalsamplePhase I: Creating a repository of adversarial/normal examplesPhase II: Generating XAI SignaturesSHAP Values(XAI signature)Phase III: Detector ConstructionXAI Signatures(adversarial/normal)input layer(n*m)1sthidden layer(dense –256)2ndhidden layer(dense -128)3rdhidden layer(dense -16)nmoutput layer(adversarial/ normal)Target Classifier𝑓(⋅)We also explore the specific detection rates of different

attack methods. In Figure 6, we present the TPR of the detector

for each attack method. The results show a high TPR for most

attacks even for a FPR of 0.05.

Training the adversarial example detector.: In each exper-

iment we generated an experiment-specific train and test sets,

by fetching the relevant SHAP values from our generated

repository. Using the SHAP values as features, we trained

a fully connected feed forward neural network with three

hidden layers (having 256, 128, 16 neurons respectively), all

with ReLU activation units and Sigmoid output. We split the

training set into train and validation subsets using a random

80/20 split and used the AdaBound optimizer [37] with default

parameters. We train for at most 500 epochs with an early stop

condition that monitors the binary cross-entropy validation loss

and decides to stop if it hasn’t improved for the last 20 epochs.

B. Results and Discussion

RQ1: baseline performance of

(i.e.,

and model pair

the detector.: For each

and

dataset

[CIFAR,ResNet56]

[MNIST,CNN]) we constructed the train and test set as fol-

lows. For the train set we used all normal and adversarial

train samples of the selected dataset – CIFAR-10 or MNIST.

Similarly, for the test set we used all normal and adversarial

test samples of the selected dataset. Each sample in the

train/test sets was represented using its SHAP values with

the class label set to be “1” for adversarial example and “0”

otherwise. In Figure 5, we present the ROC and precision-

recall curves as well as the area under those curves (AUC-ROC

and AUC-PR). As can be seen, the proposed method yields

high detection performances with AUC-ROC of 0.966/0.967

and AUC-PR of 0.958/0.961 for the CIFAR-10 and MNIST

datasets respectively.

Fig. 6. Evaluation results for RQ1 and RQ2 experiments.

RQ2: Generalization across different attack types.: This

evaluation simulates a scenario where a detector trained on

adversarial examples of known attacks is confronted with

adversarial examples generated by an unknown attack.

Given a target dataset and model, we divided each one of

its train and test sets by an (algorithm, metric) pair to get a

collection of train/test subsets. We performed random under-

sampling of each test subset to balance the number of normal

and adversarial examples in it. Then, we follow a “leave on

out” approach in which for each (algorithm,metric) pair we

train a detector on all train samples generated by all but this

pair and evaluate only on adversarial examples generated by

this pair. Similar to the previous research question, we evaluate

the general performance of our detector (Figure 5), and the

performances for each attack method separately (Figure 5).

As can be seen the overall performance in detecting unknown

attack is very similar to the case of known attacks (with

Fig. 5. ROC and PR curves of the proposed detector evaluated on CIFAR-10

and MNIST datasets.

CIFAR-10 DatasetMNIST DatasetMNIST DatasetCIFAR-10 DatasetAUC-ROC of 0.965/0.973 and AUC-PR of 0.961/0.975 for

the CIFAR-10 and MNIST datasets respectively). Similarly,

the proposed method show high performances for most attacks

even for a FPR of 0.05.

VII. CONCLUSIONS AND FUTURE WORK

The results of our experiments validate the ability of our

approach to detect adversarial examples generated by a variety

of state of the art attacks (RQ1). We showed that the detector

generalizes well when confronted with adversarial examples

generated by attacks it wasn’t train on (RQ2). These results

support our hypothesis regarding the connection between

patterns of SHAP values of the penultimate layer of the

classification model,

the distribution of the importance of

robust vs. non-robust features for the classification results and

the ability to detect adversarial examples.

Although our detection method is based on a supervised

learning model, which requires generating a big training set

of adversarial examples using various attacks, the good gen-

eralization results imply that it should be possible to devise a

semi-supervised detection approach based on the same features

to streamline the detector training process and improve its

performance further.

Our proposed method can be further extended into a generic

framework, reminiscent of antivirus or IDS systems that con-

tinuously collect and analyze benign and malicious samples,

extract signatures and, based on those signatures, classify

samples as malicious or benign or forward them for manual

analysis. A framework such as this, employing both the SHAP

based signatures discussed in this paper, and signatures used

in other, state of the art detectors, could advance the practical

ability to defend against adversarial examples.

In this paper we made a first step towards understanding the

connection between model explanations and feature robust-

ness. Rigorously studying this connection could be beneficial

both for improving the performance of our detector and for a

better understanding of the nature of adversarial examples.

Additional future work may include: (1) testing of our

method on additional datasets (from other domains) and

classification models; (2) evaluating the transferability of the

detector between underlying classification models; and (3)

evaluating our method against customized attacks adapted to

take our detector into account (which is not trivial since the

computation of SHAP values is not differentiable).

REFERENCES

[1] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow,

and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint