Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong

the direction. These generate images at a fixed L∞ dis-

tance and L2 distance from the original image, respec-

tively (modulo image box constraints). The Jacobian-

based Saliency Map Approach (JSMA) [25] greedily

modifies pixels that, based on the gradient, would most

strongly reduce the confidence of a correct classification.

This generates images at a fixed L0 distance from the

original image. DeepFool [21] extends the fast gradi-

ent methods with an iterative approach, similar to gra-

dient descent optimization. Optimization-based meth-

ods [27, 4, 19] take this approach further with more so-

phisticated optimization algorithms and specialized loss

functions.

2.2 Threat Models

In this work, we assume the adversary has full knowl-

edge of the model, including the model architecture, pa-

rameters, and the defense strategies used in the model.

Prior work has shown this assumption can often be re-

laxed. Adversarial examples generated for one model

can successfully fool other models, even ones models

of different architectures and models trained on different

data [7, 23]. An attacker can thus train a model of its own

and generate adversarial examples to fool a black-box

model [24, 19]. For simplicity we assume this stronger

white-box threat model.

Within these white-box attackers, we consider two ca-

pacities of adversaries.

Static Adversary. A static adversary is an attacker

that is not aware of any defenses that may be in place to

protect the model against adversarial examples. A static

adversary can generate adversarial examples using exist-

ing methods but does not tailor attacks to any specific

defense.

Adaptive Adversary. An adaptive adversary is an at-

tacker that is aware of the defense methods used in the

model, and can adapt attacks accordingly. This is a

strictly more powerful adversary than static adversary. In

this paper, we consider this stronger adversary.

this means that an adaptive adversary can defeat the de-

fense. To gauge how strong the combined defense is

compared to the components, we compare the level of

distortion needed to fool each (using the same optimiza-

tion method).

2.3 Problem Statement

2.4 Experimental Setup

To improve the robustness of models against adversar-

ial examples, prior work investigates into two directions.

The first direction attempts to produce correct predic-

tions on adversarial examples, while not compromising

the accuracy on legitimate inputs [26, 7, 9]. The other

(more recent) direction instead attempts to detect adver-

sarial examples, without introducing too many false pos-

itives.

In this case, the model can reject an instance

and refuse to classify those that it detects as adversar-

ial [20, 8, 30, 1].

As stated earlier, in this paper, we ask the question: if

we ensemble multiple defenses to adversarial examples,

then will the combined defense be significantly stronger

than each individual original defense? This is an im-

portant question for designing effective defense against

adversarial examples, as if the answer were “yes,” then it

may be possible to construct a strong defense simply by

ensembling multiple weaker defenses. In this paper we

argue the answer is ”no”: by considering many defenses,

we show that ensembled defenses are not always more

robust.

Defenses considered.

In this paper, we consider de-

fenses that attempt

to combine multiple (somewhat

weaker) defenses to construct a larger strong defense. In

particular, we look at three instances of ensemble de-

fense strategies. First and second are feature squeez-

ing [30] and the specialists+1 ensemble method [1],

both of which take this approach by construction. These

defenses are constructed from components that are in-

tended to be useful together. Their authors have shown

that these defenses effectively detect low-perturbation

adversarial examples generated by a static adversary.

Third, to study the effectiveness of ensembling defenses

more broadly, we merge together many detectors that

were not designed to be used in conjunction with any

other detector. In particular, as an example demonstra-

tion, we ensemble three independent detection mecha-

nisms [6, 20, 5] to build one detection mechanism.

For each of these defense strategies, we propose at-

tack methods to generate adversarial examples as an

adaptive adversary against the individual component de-

fense (when applicable) as well as the composite de-

fense strategy. We use these attack methods to evalu-

ate each component defense and composite defense: if

our method succeeds at generating adversarial examples,

Datasets and models. To evaluate the effectiveness

of the different defense strategies, we use two standard

datasets, MNIST [17] and CIFAR-10 [15] datasets.

For both datasets, we randomly sample 100 images in

the test set, filter out examples that are not correctly clas-

sified, and generate adversarial examples based on the

correctly classified images. When evaluating each de-

fense strategy, we use the same model architectures de-

scribed in their papers respectively [30, 1, 6, 20, 5].

Our experiments took up to three minutes to gener-

ate each adversarial example. The attacks we use can

scale up to larger models, which require more compu-

tation per optimization step. On the other hand, prior

work has shown that larger models are actually easier to

fool, with lower-distortion adversarial examples or better

success at a fixed level of distortion [7, 21, 28, 4]. Our

own results agree, with adversarial examples on a ResNet

for CIFAR-10 having significantly lower distortion than

adversarial examples on a smaller CNN for MNIST (a

much smaller dataset). We expect even larger datasets

would be even easier to attack.

Adversarial example generation method.

In this pa-

per, we use an optimization-based approach to generate

adversarial examples [4], which is shown to be effective

on finding adversarial examples with small distortions.

At a high level, the attack uses an optimizer to mini-

mize a loss function:

loss(x(cid:48)) = (cid:107)x(cid:48) − x(cid:107)2

2 + c · J(Fθ (x(cid:48)), y)

Here, Fθ is a part of the trained classifier that outputs

a vector of logits, and J computes some penalty based on

the logits and some label y, either a ground truth label for

non-targeted attacks or a target label for targeted attacks.

A constant c is a hyperparameter that adjusts the relative

weighting between distortion and misclassification. We

omit details of the design choice and refer the reader to

the original paper [4].

Measurement of distortion. Unless otherwise speci-

fied, we measure the distortion between an adversarial

example and the original input as the L2-norm of their

distance. Formally, d(x∗, x) = (cid:112)

i − xi)2. Each di-

mension of input images is scaled to [0, 1], i.e., 0 ≤

x∗

i , xi ≤ 1.

∑i(x∗

3 Adaptive attacks on feature squeezing

In this and next section, we investigate ensemble defense

strategies that are intentionally constructed to have com-

ponent defenses which work together to detect adver-

sarial examples. The first defense we study is feature

squeezing, proposed by Xu et al. [30, 31].

Background: feature squeezing defense. To perform

feature squeezing, one generates a lower fidelity version

of the input image through a process known as “squeez-

ing” before passing it into the classifier. Xu et al. pro-

posed two methods of squeezing:

reducing the color

depth to fewer bits, and spatially smoothing the pixels

with a median filter. According to their paper, the two

methods of squeezing work well together because they

address two major kinds of perturbation used in adver-

sarial examples: color depth reduction eliminates small

changes to many pixels, while spatial smoothing elimi-

nates large changes to a few pixels.

In order to detect adversarial examples, Xu et al. pro-

pose a system combining the two squeezing methods.

First, the system runs the classifier on three different ver-

sions of the image: the original image, the reduced-color-

depth version and the spatially smoothed version of the

original image. Then, it compares the softmax probabil-

ity vectors across these three classifier outputs. The L1

score of the input is the highest L1 distance between any

pair of softmax probability vectors among the three. It

flags inputs where the L1 score exceeds a threshold as

adversarial.

In their experiments, Xu et al. show that MNIST and

CIFAR-10 classifiers are accurate on squeezed inputs.

On adversarial examples generated by a static adversary

using FGSM [7] and JSMA [25], they show that their

detector achieves 99.74% accuracy on a test set with

equal portions benign and adversarial examples. They

also show that squeezing the input alone prevents 84 –

100% of the adversarial examples (correctly classifying

them). Recently, Xu et al. showed that a simplified de-

tector that uses the original version of the input and the

spatially-smoothed version (excluding the color-depth-

reduced version) achieves a 98.80% overall detection ac-

curacy on MNIST and 87.50% on CIFAR-10 against a

static adversary using a variety of Carlini and Wagner’s

attacks [31].

Summary of our approach and results. We demon-

strate that feature squeezing is not an effective defense in

two stages. First, we show that an adaptive attacker can

construct an adversarial example that remains adversarial

after it is squeezed by each method (color depth reduc-

tion and spatial smoothing. Then, we use this approach

to construct adversarial examples that are classified the

same way both with and without squeezing, causing the

L1 score to be smaller than a given fixed threshold. Our

results show that the combined detection method is not

effective against an adaptive attacker.

3.1 Evading individual feature squeezing

defense components

In these experiments, we evaluate whether adversarial

examples are robust to each individual feature squeezing

defense component, i.e., whether adversarial examples

remain adversarial after each individual feature squeez-

ing process (color depth reduction and spatial smooth-

ing) separately. These experiments attack the com-

ponents of the combined feature squeezing detection

scheme. Performing this attack is necessary for defeat-

ing the combined detection scheme, wherein the pre-

dicted label probabilities of squeezed images are com-

pared against each other.

3.1.1 Evading color-depth-reduction defense

The first method of squeezing an image that Xu et al. pro-

pose is color depth reduction. This method rounds each

value in the input to 2b evenly spaced values spanning

the same range, which we refer to as reducing to b bits.

Attack Approach. We use the method described in

Section 2.4 to generate adversarial examples that are ro-

bust to color depth reduction. After each step of the op-

timization procedure, an intermediate image (perturbed

from the original image) is available from the optimizer.

We check if a reduced-color-depth version of this inter-

mediate image is adversarial. We run the optimization

multiple times, initializing the optimization with random

perturbations of the original image each time, so that it

explores different optimization paths. For each original

image, we keep the successful adversarial example that

has the lowest L2 distance to the original image among

all the generated successful adversarial examples for this

original image.

Attack results on MNIST. We evaluate color depth

reduction to 1 – 7 bits. On the strongest defense eval-

uated by Xu et al., which reduces color depth to 1 bit,

we successfully generated adversarial examples for all

original images, with an average distortion of 3.86. Fig-

ure 1 shows a sample of these adversarial examples. Ta-

ble 1 summarizes our results for other bit depths. Notice

that for a system without any color depth reduction (re-

taining the original 8 bits of depth), we find adversarial

examples with an average distortion of 1.38. Reducing

color depth to fewer bits makes the system less sensitive

Figure 1: Adversarial examples for color depth reduction

(to 1 bit) on MNIST. First row: original images. Second

row: adversarially perturbed. Distortions, from left to

right: 1.49, 2.61, 2.63, 3.83, 3.89, 3.90.

Figure 2: Adversarial examples for color depth reduction

(to 3 bits) on CIFAR-10. Distortions, from left to right:

0.0194, 0.0954, 0.322, 0.942, 0.948, 0.948. Layout is the

same as Figure 1.

Bit depth Adv success Avg L2

3.86

100%

1.69

99%

1.43

100%

1.39

100%

1.44

100%

1.33

100%

1.33

100%

1.38

100%

1

2

3

4

5

6

7

8

Filter size Adv success Avg L2

1.29

100%

1.57

100%

0.612

100%

1.33

100%

1.29

100%

1.52

100%

1.51

100%

0.943

100%

0.931

100%

3 × 3

2 × 2

5 × 5

3 × 1

1 × 3

2 × 1

1 × 2

5 × 1

1 × 5

Table 1: Summary of MNIST adversarial examples that

are misclassified when reduced to different color depths.

“Adv success” measures the fraction of original images

for which we successfully found an adversarial example.

“Avg L2” measures the average L2 distortion of the suc-

cessful adversarial examples.

Table 2: Summary of MNIST adversarial examples that

are misclassified when spatially smoothed with varying

sizes of median filters. Columns have the same meaning

as in Table 1. Some filters make adversarial examples

easier to find.

to small changes, which requires larger distortions; how-

ever, the distortions are still very small.

Attack results on CIFAR-10. We evaluate color depth

reduction to 3 bits, which Xu et al. recommend as a good

balance between the accuracy on adversarial inputs and

accuracy on benign images for CIFAR-10. We succeeded

at generating adversarial examples for all original im-

ages, with an average distortion of 0.945. Figure 2 shows

a sample of these adversarial examples. For comparison,

adversarial examples for a classifier without color depth

reduction have an average distortion of 0.214. Although

this method of squeezing increases the distortion needed

for successfully generating non-targeted adversarial ex-

amples using the same optimization method, again, such

a distortion is still small and imperceptible.

Summary. An adaptive attacker can successfully gen-

erate adversarial examples with small distortions for a

system that applies color depth reduction to the input im-

age before classifying it.

3.1.2 Evading spatial smoothing

Xu et al. propose a second method for feature squeez-

ing, which applies a median filter to the input, which re-

places each pixel with the median value of a neighbor-

hood around the pixel.

To generate adversarial examples that are misclassi-

fied after spatial smoothing, we use the procedure from

Section 2.4 with the addition of a median filter as part of

the classification model. A median filter for TensorFlow

was not available, so we implemented our own.

Attack results on MNIST. We evaluate a range of me-

dian filter sizes, ranging from 1 × 2 to 5 × 5. For a 3 × 3

filter, with which Xu et al. achieved the best accuracy, we

successfully generated adversarial examples for all orig-

inal images, with an average distortion of 1.29. Figure 3

shows a sample of these adversarial examples. Table 2

summarizes our results for other filter sizes. Larger me-

dian filters did not require greater distortion. Compared

to adversarial examples generated for a system without

any spatial smoothing (average distortion of 1.38), the

average distortion is not increased.

In particular,

Background: Composite feature squeezing defense

The detection scheme combines both methods of squeez-

ing.

the detection system has three

“branches,” where each one runs the classifier on a dif-

ferent version of the input, the original input, a reduced-

color-depth version and a spatially-smoothed version of

the original input. These three branches output different

softmax probability vectors, and the scheme compares

the L1 distance between these vectors against a threshold

to determine whether the input is adversarial.

In this experiment, we evaluate the effectiveness of

the combined defense against an adaptive attacker, i.e.,

whether adversarial examples can be misclassified and

bypass this detection scheme.

Attack Approach. We use the procedure from Sec-

tion 3.1.1 with modifications to support the detection

mechanism. In this attack, we run two instances of the

model in parallel during optimization, representing the

detector branches that operate on the input and the spa-

tially smoothed version of the input. The optimizer re-

ceives gradients from both models, equally. We do not

include the reduced-color-depth branch in the gradient

calculation, because the color depth reduction removes

gradient information; it is, however, included when we

compute the L1 score. We collect only adversarial exam-

ples that have an L1 score below a threshold of 0.3076,

a level at which Xu et al. achieved the best accuracy in

their experiments on MNIST.

Attack results on MNIST We evaluate a combination

of color depth reduction to 1 bit and smoothing with a

2 × 2 median filter, which Xu et al. found to be accu-

rate on adversarial examples generated by a static adver-

sary [31]. We successfully generated adversarial exam-

ples for all original images, with an average distortion of

4.76 and L1 score of 0.209. Figure 5 shows a sample of

these adversarial examples. These examples are misclas-

sified and successfully evade detection. This distortion

is 23.3% larger than for color depth reduction alone, but

still very small.

Attack results on CIFAR-10. We evaluate a combi-

nation of color depth reduction to 3 bits and smoothing

with a 2 × 2 median filter, a combination of settings that

perform well in Xu et al.’s experiments. We successfully

generated adversarial examples for all original images,

with an average distortion of 0.601 and L1 score of 0.168.

Figure 6 shows a sample of these adversarial examples.

These examples are misclassified and successfully evade

detection. This distortion is even lower than that of the

color depth reduction defense alone. Although Xu et al.

do not prescribe a threshold specific to CIFAR-10, the

Figure 3: Adversarial examples for spatial smoothing

(with 3 × 3 filter) on MNIST. Distortions, from left to

right: 0.236, 0.241, 0.282, 1.27, 1,31, 1.31. Layout is the

same as Figure 1.

Figure 4: Adversarial examples for spatial smoothing

(with 2 × 2 filter) on CIFAR-10. Distortions, from left

to right: 0.0273, 0.0537, 0.0584, 0.198, 0.211, 0.212.

Layout is the same as Figure 1.

Attack results on CIFAR-10 We evaluate a 2 × 2 me-

dian filter, which Xu et al. identify as achieving a good

rejection rate of adversarial examples and accuracy on

benign images on CIFAR-10. We successfully generated

adversarial examples for all original images, which have

an average distortion of 0.205. Figure 4 shows a sample

of these adversarial examples. The average distortion is

not higher than for a system without spatial smoothing

(0.214).

Summary Spatial smoothing alone is not an effective

defense against an adaptive attacker. We have shown that

an adaptive adversary can create adversarial examples for

a system that applies spatial smoothing which are not

more distorted than adversarial examples for a baseline

system that does not apply spatial smoothing.

3.2 Evading

combination of multiple

squeezing techniques

While the individual feature squeezing techniques are

weak against an adaptive attacker, we investigate

whether the detection scheme that combines them is

stronger. In this case, we find that this detection scheme

is not much stronger than the strongest component de-

fense, color depth reduction.

Figure 5: Adversarial examples for combined feature

squeezing detection on MNIST. Distortions, from left to

right: 2.00, 2.04, 2.39, 4.66, 4.77, 4.79. Layout is the

same as Figure 1.

Figure 7: Adversarial examples for specialists+1 on

MNIST. Distortions, from left to right: 1.55, 1.76, 1.83,

3.77, 3.90, 3.93. Layout is the same as Figure 1.

the set of classes with which i is most often confused in

adversarial examples. To compute Ui, Abbasi and Gagn´e

select the top 80% of misclassifications caused by non-

targeted FGSM attacks for each class i. Further, K addi-

tional subsets are defined: UK+i = C \ Ui to be the com-

plement set of Ui. For each j = 1, ..., 2K, a specialist

classifier Fj is trained on a subset of the dataset contain-

ing images belonging to the classes in U j to classify input

images into the classes in U j only. In addition, a general-

ist classifier F2K+1 is trained to classify input images into

classes in U2K+1 = C. Each classifier in the ensemble

may be susceptible to basic adversarial examples, but the

proposed defense assumes that each specialist can detect

a few specific attacks, thus the attacker cannot fool all

specialists and the generalist at the same time. The de-

fense combines them to jointly detect general adversarial

examples.

In order to classify an input, the system first checks if,

for any class i, the generalist classifier and all specialists

that can classify i agree that the input belongs to class i.

If such a class i exists, note that at most one class can

get the generalist’s vote, it must be unique. In this case,

the system takes the mean of the outputs of the generalist

and the specialists that can classify i. Otherwise, at least

one classifier has misclassified the input, and the system

takes the mean of the outputs of all classifiers in the en-

semble.

Abbasi and Gagn´e [1] find that using an ensemble con-

structed this way successfully reduces the system’s con-

fidence (mean confidence among classifiers activated by

the voting scheme) on adversarial examples generated by

a static attacker using FGSM [7], DeepFool [21], and

Szegedy et al.’s approach [27]. They conclude that a

classification system can use an ensemble of diverse spe-

cialists this way and detect low-confidence examples as

adversarial.

Attack approach.

In this experiment, we evaluate the

effectiveness of Abbasi and Gagn´e’s specialists+1 en-

semble against an adaptive attacker. We considered a

scenario where a user provides an image to a system, and

Figure 6: Adversarial examples for combined feature

squeezing detection on CIFAR-10. Distortions, from left

to right: 0.117, 0.120, 0.130, 0.604, 0.614, 0.617. Layout

is the same as Figure 1.

average L1 score for these examples is lower (i.e., de-

tected as less adversarial) than the average L1 score for

the original images, which is 0.225.

Summary. The detection scheme that combines two

methods of squeezing is not always stronger than the

strongest component, color depth reduction. The im-

provement is low even on MNIST, which is particu-

larly well suited for feature squeezing, with images be-

ing black and white (little change from color depth re-

duction) and having large, contiguous areas of the same

color (little change from spatial smoothing). On CIFAR-

10, the combined attack requires less distortion than the

color depth reduction defense alone.

4 Evading ensemble of specialists

We study a second defense that combines multiple com-

ponent defenses, an ensemble of specialists, proposed by

Abbasi and Gagn´e [1].

Background: ensemble of specialist defense. The de-

fense consists of a generalist classifier (which classifies

among all classes) and a collection of specialists (which

classify among subsets of the classes). The specialists

classify subsets of the classes as follows. Where C is the

set of all K classes in the task, for each class i, let Ui be

the system uses a specialists+1 ensemble to classify the

image or reject it as adversarial.

We attempt to create targeted adversarial examples,

where we chose target classes randomly. For each origi-

nal image, then our goal is to create an adversarial exam-

ple that is classified as the target class by the generalist

classifier and all applicable specialists at the same time,

and with high confidence from those classifiers. We use

the procedure from Section 2.4 to generate adversarial

examples. In this experiment, we kept only adversarial

examples that were misclassified with confidence greater

than the average confidence on a sample of benign im-

ages, 0.999708. We modified the loss function to support

multiple classifiers:

loss(x(cid:48)) = (cid:107)x(cid:48) − x(cid:107)2

2 + c∑ j∈{1,...,2K+1};y∗∈U j

J(Fj(x(cid:48)), y∗)

We evaluate this defense on MNIST only. While Ab-

basi and Gagn´e also propose the defense for CIFAR-10,

the architecture described in their experiments have low

accuracy on CIFAR-10, resulting in low confidence even

in benign images.

Attack results on MNIST. We successfully generated

adversarial examples for all original images, which have

an average distortion of 3.87. Figure 7 shows a sample

of these adversarial examples in the second row. These

adversarial examples are classified as the target label by

the generalist and all applicable specialists. For compar-

ison, the average confidence of a single generalist clas-

sifier on correctly classified benign images is 0.998951,

and a batch of targeted adversarial examples with at least

that confidence has average distortion 3.65. The distor-

tion needed for high-confidence adversarial examples on

specialists+1 is 6.03% higher than for a non-ensemble

MNIST classifier.

Although this defense defines the specialists to focus

on common misclassifications caused by non-targeted

adversarial examples, it is still weaker at detecting the

common misclassifications. Among the examples, 33

targeted a class that the original image’s ground truth

class was commonly confused with. The average dis-

tortion for these images is 3.06, below the average of the

entire set.

Source Defense

Gong Metzen

Feinman

t Gong

e

g

r

a

T

Metzen

Feinman

100% 51%

100%

43%

92%

96%

21%

18%

100%

Table 3: Probability that adversarial examples con-

structed for a given source defense also fool the given

target defense on CIFAR-10. Defenses generated against

Metzen et al. transfer to the others with the highest prob-

ability, and Feinman et al. with lowest.

5 Evading ensemble of detectors

In the previous sections, we have investigated ensembles

of defenses that are intentionally constructed to be useful

together. In Xu et al.’s work, the color depth reduction is

intended to remove small changes to many pixels, and

the median smoothing to remove large changes to a few

pixels. Similarly, Abbasi and Gagn´e propose using an

ensemble of generalist and specialist classifiers together;

without the others, this approach would not be useful.

To study the effectiveness of ensembling defenses

more broadly, we merge together three recently proposed

detectors that were not designed to be used in conjunc-

tion with any other detector. We consider only detectors

that are applied to a fixed classification network for sim-

plicity, and therefore study the following schemes:

• Gong et al. propose using adversarial training to

detect adversarial examples [6]. Given the original

model, generate adversarial examples on the train-

ing data. Then, train a new classifier that distin-

guishes the original training data from the adversar-

ial data.

• Metzen et al. construct a similar scheme, however

instead of using the original images as the input to

the detector, they train on the inner convolutional

layers of the network [20].

• Feinman et al. examine the final hidden layer of a

neural network and find that adversarial examples

are separable from the original images by training a

density estimate using Gaussian kernels [5].

Summary. The specialists+1 ensemble does not ef-

fectively ensure low confidence on adversarial exam-

ples generated by an a...