]

[

Adversarial Examples Are Not Bugs, They Are Features

Andrew Ilyas∗

MIT

[email protected]

Shibani Santurkar∗

MIT

[email protected]

Dimitris Tsipras∗

MIT

[email protected]

Logan Engstrom∗

MIT

[email protected]

Brandon Tran

MIT

[email protected]

Aleksander M ˛adry

MIT

[email protected]

Abstract

Adversarial examples have attracted signiﬁcant attention in machine learning, but the reasons for their

existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly at-

tributed to the presence of non-robust features: features (derived from patterns in the data distribution) that

are highly predictive, yet brittle and (thus) incomprehensible to humans. After capturing these features

within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we

present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalign-

ment between the (human-speciﬁed) notion of robustness and the inherent geometry of the data.

1 Introduction

The pervasive brittleness of deep neural networks [Sze+14; Eng+19b; HD19; Ath+18] has attracted signif-

icant attention in recent years. Particularly worrisome is the phenomenon of adversarial examples [Big+13;

Sze+14], imperceptibly perturbed natural inputs that induce erroneous predictions in state-of-the-art clas-

siﬁers. Previous work has proposed a variety of explanations for this phenomenon, ranging from theoreti-

cal models [Sch+18; BPR18] to arguments based on concentration of measure in high-dimensions [Gil+18;

MDM18; Sha+19a]. These theories, however, are often unable to fully capture behaviors we observe in

practice (we discuss this further in Section 5).

More broadly, previous work in the ﬁeld tends to view adversarial examples as aberrations arising either

from the high dimensional nature of the input space or statistical ﬂuctuations in the training data [Sze+14;

GSS15; Gil+18]. From this point of view, it is natural to treat adversarial robustness as a goal that can

be disentangled and pursued independently from maximizing accuracy [Mad+18; SHS19; Sug+19], ei-

ther through improved standard regularization methods [TG16] or pre/post-processing of network in-

puts/outputs [Ues+18; CW17a; He+17].

In this work, we propose a new perspective on the phenomenon of adversarial examples. In contrast

to the previous models, we cast adversarial vulnerability as a fundamental consequence of the dominant

supervised learning paradigm. Speciﬁcally, we claim that:

Adversarial vulnerability is a direct result of our models’ sensitivity to well-generalizing features in the data.

Recall that we usually train classiﬁers to solely maximize (distributional) accuracy. Consequently, classiﬁers

tend to use any available signal to do so, even those that look incomprehensible to humans. After all, the

presence of “a tail” or “ears” is no more natural to a classiﬁer than any other equally predictive feature. In

fact, we ﬁnd that standard ML datasets do admit highly predictive yet imperceptible features. We posit that

∗Equal contribution

our models learn to rely on these “non-robust” features, leading to adversarial perturbations that exploit

this dependence.1

Our hypothesis also suggests an explanation for adversarial transferability: the phenomenon that adver-

sarial perturbations computed for one model often transfer to other, independently trained models. Since

any two models are likely to learn similar non-robust features, perturbations that manipulate such fea-

tures will apply to both. Finally, this perspective establishes adversarial vulnerability as a human-centric

phenomenon, since, from the standard supervised learning point of view, non-robust features can be as

important as robust ones. It also suggests that approaches aiming to enhance the interpretability of a given

model by enforcing “priors” for its explanation [MV15; OMS17; Smi+17] actually hide features that are

“meaningful” and predictive to standard models. As such, producing human-meaningful explanations that

remain faithful to underlying models cannot be pursued independently from the training of the models

themselves.

To corroborate our theory, we show that it is possible to disentangle robust from non-robust features in

standard image classiﬁcation datasets. Speciﬁcally, given any training dataset, we are able to construct:

1. A “robustiﬁed” version for robust classiﬁcation (Figure 1a)2 . We demonstrate that it is possible to

effectively remove non-robust features from a dataset. Concretely, we create a training set (seman-

tically similar to the original) on which standard training yields good robust accuracy on the original,

unmodiﬁed test set. This ﬁnding establishes that adversarial vulnerability is not necessarily tied to the

standard training framework, but is also a property of the dataset.

2. A “non-robust” version for standard classiﬁcation (Figure 1b)2. We are also able to construct a

training dataset for which the inputs are nearly identical to the originals, but all appear incorrectly

labeled. In fact, the inputs in the new training set are associated to their labels only through small

adversarial perturbations (and hence utilize only non-robust features). Despite the lack of any predictive

human-visible information, training on this dataset yields good accuracy on the original, unmodiﬁed

test set. This demonstrates that adversarial perturbations can arise from ﬂipping features in the data

that are useful for classiﬁcation of correct inputs (hence not being purely aberrations).

Finally, we present a concrete classiﬁcation task where the connection between adversarial examples and

non-robust features can be studied rigorously. This task consists of separating Gaussian distributions, and

is loosely based on the model presented in Tsipras et al. [Tsi+19], while expanding upon it in a few ways.

First, adversarial vulnerability in our setting can be precisely quantiﬁed as a difference between the intrinsic

data geometry and that of the adversary’s perturbation set. Second, robust training yields a classiﬁer which

utilizes a geometry corresponding to a combination of these two. Lastly, the gradients of standard models

can be signiﬁcantly more misaligned with the inter-class direction, capturing a phenomenon that has been

observed in practice in more complex scenarios [Tsi+19].

2 The Robust Features Model

We begin by developing a framework, loosely based on the setting proposed by Tsipras et al. [Tsi+19],

that enables us to rigorously refer to “robust” and “non-robust” features. In particular, we present a set of

deﬁnitions which allow us to formally describe our setup, theoretical results, and empirical evidence.

Setup. We consider binary classiﬁcation3, where input-label pairs (x, y)

(data) distribution

to a given input x.

; the goal is to learn a classiﬁer C :

X → {±

}

are sampled from a

}

which predicts a label y corresponding

∈ X × {±

1It is worth emphasizing that while our ﬁndings demonstrate that adversarial vulnerability does arise from non-robust features, they

do not preclude the possibility of adversarial vulnerability also arising from other phenomena [TG16; Sch+18]. For example, Nakkiran

[Nak19a] constructs adversarial examples that do not exploit non-robust features (and hence do not allow one to learn a generalizing

model from them). Still, the mere existence of useful non-robust features sufﬁces to establish that without explicitly discouraging

models from utilizing these features, adversarial vulnerability will remain an issue.

2The corresponding datasets for CIFAR-10 are publicly available at http://git.io/adv-datasets.

3Our framework can be straightforwardly adapted though to the multi-class setting.

(a)

(b)

Figure 1: A conceptual diagram of the experiments of Section 3. In (a) we disentangle features into combi-

nations of robust/non-robust features (Section 3.1). In (b) we construct a dataset which appears mislabeled

to humans (via adversarial examples) but results in good accuracy on the original test set (Section 3.2).

We deﬁne a feature to be a function mapping from the input space

to the real numbers, with the set

are

of all features thus being

shifted/scaled to be mean-zero and unit-variance (i.e., so that E

[ f (x)2] = 1),

in order to make the following deﬁnitions scale-invariant4. Note that this formal deﬁnition also captures

what we abstractly think of as features (e.g., we can construct an f that captures how “furry” an image is).

. For convenience, we assume that the features in

[ f (x)] = 0 and E

X →

(x,y)

∼D

{

}

Useful, robust, and non-robust features. We now deﬁne the key concepts required for formulating our

framework. To this end, we categorize features in the following manner:

•

ρ-useful features: For a given distribution

the true label in expectation, that is if

, we call a feature f ρ-useful (ρ > 0) if it is correlated with

(x,y)

∼D

f (x)]

ρ.

≥

(1)

We then deﬁne ρ

if a feature f is negatively correlated with the label, then

classiﬁer trained on ρ-useful features can attain non-trivial generalization performance.

( f ) as the largest ρ for which feature f is ρ-useful under distribution

. (Note that

f is useful instead.) Crucially, a linear

−

•

( f ) > 0). We refer to f as a

γ-robustly useful features: Suppose we have a ρ-useful feature f (ρ

robust feature (formally a γ-robustly useful feature for γ > 0) if, under adversarial perturbation (for

some speciﬁed set of valid perturbations ∆), f remains γ-useful. Formally, if we have that

(x,y)

inf

∆(x)

∈

∼D (cid:20)

f (x + δ)

γ.

≥

(cid:21)

(2)

•

Useful, non-robust features: A useful, non-robust feature is a feature which is ρ-useful for some ρ

bounded away from zero, but is not a γ-robust feature for any γ

0. These features help with classi-

ﬁcation in the standard setting, but may hinder accuracy in the adversarial setting, as the correlation

with the label can be ﬂipped.

≥

Classiﬁcation.

weight vector w, and a scalar bias b. For a given input x, the classiﬁer predicts the label y as

In our framework, a classiﬁer C = (F, w, b) is comprised of a set of features F

, a

⊆ F

C(x) = sgn

b + ∑

∈

(cid:32)

w f ·

f (x)

(cid:33)

For convenience, we denote the set of features learned by a classiﬁer C as FC.

4This restriction can be straightforwardly removed by simply shifting/scaling the deﬁnitions.

Robust datasetTraingood standard accuracy good robust accuracygood standard accuracy bad robust accuracyUnmodiﬁed test setTraining imagefrogfrogfrogNon-robust datasetTrainEvaluate on original test setTraining imageRobust Features: dog Non-Robust Features: dogdogRelabel as catRobust Features: dog Non-Robust Features: catcatcatmax P(cat) Adversarial example towards “cat” Traingood accuracyStandard Training. Training a classiﬁer is performed by minimizing a loss function (via empirical risk

minimization (ERM)) that decreases with the correlation between the weighted combination of the features

and the label. The simplest example of such a loss is 5

(x,y)

[

Lθ(x, y)] =

−

∼D

(x,y)

· (cid:32)

∼D (cid:34)

b + ∑

∈

w f ·

f (x)

(cid:33)(cid:35)

(3)

When minimizing classiﬁcation loss, no distinction exists between robust and non-robust features: the only

distinguishing factor of a feature is its ρ-usefulness. Furthermore, the classiﬁer will utilize any ρ-useful

feature in F to decrease the loss of the classiﬁer.

In the presence of an adversary, any useful but non-robust features can be made anti-

Robust training.

correlated with the true label, leading to adversarial vulnerability. Therefore, ERM is no longer sufﬁcient

to train classiﬁers that are robust, and we need to explicitly account for the effect of the adversary on the

classiﬁer. To do so, we use an adversarial loss function that can discern between robust and non-robust

features [Mad+18]:

(x,y)

∆(x) Lθ(x + δ, y)

max

∈

(cid:21)

∼D (cid:20)

(4)

for an appropriately deﬁned set of perturbations ∆. Since the adversary can exploit non-robust features to

degrade classiﬁcation accuracy, minimizing this adversarial loss (as in adversarial training [GSS15; Mad+18])

can be viewed as explicitly preventing the classiﬁer from learning a useful but non-robust combination of

features.

Remark. We want to note that even though the framework above enables us to formally describe and

predict the outcome of our experiments, it does not necessarily capture the notion of non-robust features

exactly as we intuitively might think of them. For instance, in principle, our theoretical framework would

allow for useful non-robust features to arise as combinations of useful robust features and useless non-

robust features [Goh19b]. These types of constructions, however, are actually precluded by our experi-

mental results (in particular, the classiﬁers trained in Section 3 would not generalize). This shows that our

experimental ﬁndings capture a stronger, more ﬁne-grained statement than our formal deﬁnitions are able

to express. We view bridging this gap as an interesting direction for future work.

3 Finding Robust (and Non-Robust) Features

The central premise of our proposed framework is that there exist both robust and non-robust features that

constitute useful signals for standard classiﬁcation. We now provide evidence in support of this hypothesis

by disentangling these two sets of features.

On one hand, we will construct a “robustiﬁed” dataset, consisting of samples that primarily contain

robust features. Using such a dataset, we are able to train robust classiﬁers (with respect to the standard

test set) using standard (i.e., non-robust) training. This demonstrates that robustness can arise by removing

certain features from the dataset (as, overall, the new dataset contains less information about the original

training set). Moreover, it provides evidence that adversarial vulnerability is caused by non-robust features

and is not inherently tied to the standard training framework.

On the other hand, we will construct datasets where the input-label association is based purely on non-

robust features (and thus the corresponding dataset appears completely mislabeled to humans). We show

that this dataset sufﬁces to train a classiﬁer with good performance on the standard test set. This indicates

that natural models use non-robust features to make predictions, even in the presence of robust features.

These features alone are actually sufﬁcient for non-trivial generalizations performance on natural images,

which indicates that they are indeed valuable features, rather than artifacts of ﬁnite-sample overﬁtting.

A conceptual description of these experiments can be found in Figure 1.

5Just as for the other parts of this model, we use this loss for simplicity only—it is straightforward to generalize to more practical

loss function such as logistic or hinge loss.

(a)

(b)

Figure 2: Left: Random samples from our variants of the CIFAR-10 [Kri09] training set: the original training

set; the robust training set

DR, restricted to features used by a robust model; and the non-robust training

set

DNR, restricted to features relevant to a standard model (labels appear incorrect to humans). Right:

(cid:98)

) for models trained with: (i) standard training

Standard and robust accuracy on the CIFAR-10 test set (

(cid:98)

(on

DR.

Models trained on

DNR reﬂect the original models used to create them: notably, standard training

(cid:98)

DR yields nontrivial robust accuracy. Results for Restricted-ImageNet [Tsi+19] are in D.8 Figure 12.

(cid:98)

DNR; (iii) adversarial training (on

(cid:98)

); and (iv) standard training on

) ; (ii) standard training on

DR and

(cid:98)

3.1 Disentangling robust and non-robust features

Recall that the features a classiﬁer learns to rely on are based purely on how useful these features are

for (standard) generalization. Thus, under our conceptual framework, if we can ensure that only robust

features are useful, standard training should result in a robust classiﬁer. Unfortunately, we cannot directly

manipulate the features of very complex, high-dimensional datasets. Instead, we will leverage a robust

model and modify our dataset to contain only the features that are relevant to that model.

In terms of our formal framework (Section 2), given a robust (i.e., adversarially trained [Mad+18]) model

C we aim to construct a distribution

(x,y)

∼

DR which satisﬁes:

(cid:98)

[ f (x)

y] =

(x,y)

(cid:40)

∼D

[ f (x)

if f

∈

otherwise,

(5)

(cid:98)

We will construct a training set for

where FC again represents the set of features utilized by C. Conceptually, we want features used by C to

be as useful as they were on the original distribution

while ensuring that the rest of the features are not

useful under

xr from the original training set for

DR via a one-to-one mapping x

. In the case of a deep neural network, FC corresponds to exactly the set of activations in the penultimate

layer (since these correspond to inputs to a linear classiﬁer). To ensure that features used by the model are

(cid:98)

equally useful under both training sets, we (approximately) enforce all features in FC to have similar values

for both x and xr through the following optimization:

DNR.

(cid:98)

(cid:55)→

where x is the original input and g is the mapping from x to the representation layer. We optimize this

objective using gradient descent in input space6.

min

xr (cid:107)

g(xr)

g(x)

(cid:107)2,

−

(6)

Since we don’t have access to features outside FC, there is no way to ensure that the expectation in (5) is

FC. To approximate this condition, we choose the starting point of gradient descent for the

zero for all f

optimization in (6) to be an input x0 which is drawn from

independently of the label of x (we also explore

sampling x0 from noise in Appendix D.1). This choice ensures that any feature present in that input will

(cid:54)∈

6We follow [Mad+18] and normalize gradient steps during this optimization. Experimental details are provided in Appendix C.

“airplane’’“ship’’“dog’’“frog’’“truck’’D!DNR!DRStd Training using Adv Training using Std Training using RStd Training using NR020406080100Test Accuracy on (%)Std accuracyAdv accuracy (=0.25)not be useful since they are not correlated with the label in expectation over x0. The underlying assumption

here is that, when performing the optimization in (6), features that are not being directly optimized (i.e.,

features outside FC) are not affected. We provide pseudocode for the construction in Figure 5 (Appendix C).

DR (a few random samples are visualized in Figure 2a), we train a clas-

). The

siﬁer using standard (non-robust) training. We then test this classiﬁer on the original test set (i.e.

results (Figure 2b) indicate that the classiﬁer learned using the new dataset attains good accuracy in both

(cid:98)

standard and adversarial settings 7 8.

Given the new training set for

As a control, we repeat this methodology using a standard (non-robust) model for C in our construction

DNR are shown in Figure 2a—they

of the dataset. Sample images from the resulting “non-robust dataset”

tend to resemble more the source image of the optimization x0 than the target image x. We ﬁnd that training

on this dataset leads to good standard accuracy, yet yields almost no robustness (Figure 2b). We also verify

(cid:98)

that this procedure is not simply a matter of encoding the weights of the original model—we get the same

results for both

DNR if we train with different architectures than that of the original models.

(cid:98)

Overall, our ﬁndings corroborate the hypothesis that adversarial examples can arise from (non-robust)

features of the data itself. By ﬁltering out non-robust features from the dataset (e.g. by restricting the set of

available features to those used by a robust model), one can train a signiﬁcantly more robust model using

standard training.

DR and

(cid:98)

3.2 Non-robust features sufﬁce for standard classiﬁcation

The results of the previous section show that by restricting the dataset to only contain features that are used

by a robust model, standard training results in classiﬁers that are signiﬁcantly more robust. This suggests

that when training on the standard dataset, non-robust features take on a large role in the resulting learned

classiﬁer. Here we set out to show that this role is not merely incidental or due to ﬁnite-sample overﬁtting.

In particular, we demonstrate that non-robust features alone sufﬁce for standard generalization— i.e., a

model trained solely on non-robust features can perform well on the standard test set.

To show this, we construct a dataset where the only features that are useful for classiﬁcation are non-

robust features (or in terms of our formal model from Section 2, all features f that are ρ-useful are non-

robust). To accomplish this, we modify each input-label pair (x, y) as follows. We select a target class t

either (a) uniformly at random among classes (hence features become uncorrelated with the labels) or (b)

deterministically according to the source class (e.g. using a ﬁxed permutation of labels). Then, we add a

small adversarial perturbation to x in order to ensure it is classiﬁed as t by a standard model. Formally:

xadv = arg min

(cid:107)≤

(cid:107)

x(cid:48)−

LC(x(cid:48), t),

(7)

where LC is the loss under a standard (non-robust) classiﬁer C and ε is a small constant. The resulting

inputs are nearly indistinguishable from the originals (Appendix D Figure 9)—to a human observer, it thus

appears that the label t assigned to the modiﬁed input is simply incorrect. The resulting input-label pairs

(xadv, t) make up the new training set (pseudocode in Appendix C Figure 6).

xadv −

Now, since

is small, by deﬁnition the robust features of xadv are still correlated with class

y (and not t) in expectation over the dataset. After all, humans still recognize the original class. On the

other hand, since every xadv is strongly classiﬁed as t by a standard classiﬁer, it must be that some of the

non-robust features are now strongly correlated with t (in expectation).

(cid:107)

In the case where t is chosen at random, the robust features are originally uncorrelated with the label t (in

expectation), and after the adversarial perturbation can be only slightly correlated (hence being signiﬁcantly

7In an attempt to explain the gap in accuracy between the model trained on

DR and the original robust classiﬁer C, we test

distributional shift, by reporting results on the “robustiﬁed” test set in Appendix D.3.

(cid:98)

8In order to gain more conﬁdence in the robustness of the resulting model, we attempt several diverse attacks in Appendix D.2.

less useful for classiﬁcation than before) 9. Formally, we aim to construct a dataset

(x,y)

f (x)]

Drand

∼

> 0

if f non-robustly useful under

0 otherwise.

(cid:40)

(cid:39)

(cid:98)

In contrast, when t is chosen deterministically based on y, the robust features actually point away from

the assigned label t. In particular, all of the inputs labeled with class t exhibit non-robust features correlated

with t, but robust features correlated with the original class y. Thus, robust features on the original training

set provide signiﬁcant predictive power on the training set, but will actually hurt generalization on the

standard test set. Viewing this case again using the formal model, our goal is to construct

Drand where 10 :

(cid:98)

(8)

(x,y)

Ddet

∼

(cid:98)

f (x)] 



> 0

< 0

if f non-robustly useful under

if f robustly useful under

R otherwise ( f not useful under

∈

)11

Ddet such that

(cid:98)

(9)

We ﬁnd that standard training on these datasets actually generalizes to the original test set, as shown in

Table 1). This indicates that non-robust features are indeed useful for classiﬁcation in the standard setting.

Ddet (where all the robust features are correlated with the wrong class), results

Remarkably, even training on

in a well-generalizing classiﬁer. This indicates that non-robust features can be picked up by models during

standard training, even in the presence of robust features that are predictive 1213.

(cid:98)



Source Dataset

Dataset

CIFAR-10

ImageNetR

95.3%

63.3%

43.7%

96.6%

87.9%

64.4%

Drand

Ddet

(cid:98)

Figure 3: Transfer rate of adversarial exam-

ples from a ResNet-50 to different architectures

alongside test set performance of these archi-

tecture when trained on the dataset generated

in Section 3.2. Architectures more susceptible

to transfer attacks also performed better on the

standard test set supporting our hypothesis that

adversarial transferability arises from utilizing

similar non-robust features.

Table 1: Test accuracy (on

) of classiﬁers

Ddet training sets

Drand, and

trained on the

created using a standard (non-robust) model.

(cid:98)

For both

Ddet, only non-robust fea-

Drand and

tures correspond to useful features on both the

. These datasets are constructed

train set and

(cid:98)

using adversarial perturbations of x towards a

class t (random for

Drand and deterministic for

Ddet); the resulting images are relabeled as t.

(cid:98)

3.3 Transferability can arise from non-robust features

One of the most intriguing properties of adversarial examples is that they transfer across models with dif-

ferent architectures and independently sampled training sets [Sze+14; PMG16; CRP19]. Here, we show

9Goh [Goh19a] provides an approach to quantifying this “robust feature leakage” and ﬁnds that one can obtain a (small) amount

of test accuracy by leveraging robust feature leakage on

Drand.

models to simulate access to robust and non-robust features.

(cid:98)

10Note that the optimization procedure we describe aims to merely approximate this condition, where we once again use trained

11 Note that regardless how useful a feature is on

unaltered test set.

12Additional results and analysis (e.g. training curves, generating

13We also show that the models trained on

Ddet, since it is useless on

(cid:98)

Drand and

(cid:98)

it cannot provide any generalization beneﬁt on the

Ddet with a robust model, etc.) are in App. D.6 and D.5

(cid:98)

Ddet generalize to CIFAR-10.1 [Rec+19] in Appendix D.7.

(cid:98)

253035404550Test accuracy (%; trained on Dy+1)60708090100Transfer success rate (%)VGG-16Inception-v3ResNet-18DenseNetResNet-50that this phenomenon can in fact be viewed as a natural consequence of the existence of non-robust fea-

tures. Recall that, according to our main thesis, adversarial examples can arise as a result of perturbing

well-generalizing, yet brittle features. Given that such features are inherent to the data distribution, differ-

ent classiﬁers trained on independent samples from that distribution are likely to utilize similar non-robust

features. Consequently, an adversarial example constructed by exploiting the non-robust features learned

by one classiﬁer will transfer to any other classiﬁer utilizing these features in a similar manner.

In order to illustrate and corroborate this hypothesis, we train ﬁve different architectures on the dataset

generated in Section 3.2 (adversarial examples with deterministic labels) for a standard ResNet-50 [He+16].

Our hypothesis would suggest that architectures which learn better from this training set (in terms of per-

formance on the standard test set) are more likely to learn similar non-robust features to the original clas-

siﬁer. Indeed, we ﬁnd that the test accuracy of each architecture is predictive of how often adversarial

examples transfer from the original model to standard classiﬁers with that architecture (Figure 3). In a sim-

ilar vein, Nakkiran [Nak19a] constructs a set of adversarial perturbations that is explicitly non-transferable

and ﬁnds that these perturbations cannot be used to learn a good classiﬁer. These ﬁndings thus corrobo-

rate our hypothesis that adversarial transferability arises when models learn similar brittle features of the

underlying dataset.

4 A Theoretical Framework for Studying (Non)-Robust Features

The experiments from the previous section demonstrate that the conceptual framework of robust and non-

robust features is strongly predictive of the empirical behavior of state-of-the-art models on real-world

datasets. In order to further strengthen our understanding of the phenomenon, we instantiate the frame-

work in a concrete setting that allows us to theoretically study various properties of the corresponding

model. Our model is similar to that of Tsipras et al. [Tsi+19] in the sense that it contains a dichotomy

between robust and non-robust features, but extends upon it in a number of ways:

1. The adversarial vulnerability can be explicitly expressed as a difference between the inherent data

metric and the (cid:96)2 metric.

2. Robust learning corresponds exactly to learning a combination of these two metrics.

3. The gradients of adversarially trained models align better with the adversary’s metric.

Setup. We study a simple problem of maximum likelihood classiﬁcation between two Gaussian distributions.

In particular, given samples (x, y) sampled from

according to

our goal is to learn parameters Θ = (µ, Σ) such that

u.a.r.

∼ {−

1, +1

}

∗

, Σ

∗

∼ N

Θ = arg min

µ,Σ

(x,y)

∼D

[(cid:96)(x; y

µ, Σ)] ,

(10)

(11)

where (cid:96)(x; µ, Σ) represents the Gaussian negative log-likelihood (NLL) function. Intuitively, we ﬁnd the

parameters µ, Σ which maximize the likelihood of the sampled data under the given model. Classiﬁcation

under this model can be accomplished via likelihood test: given an unlabeled sample x, we predict y as

y = arg max

(cid:96)(x; y

µ, Σ) = sign

x(cid:62)Σ−

1µ

In turn, the robust analogue of this problem arises from replacing (cid:96)(x; y

perturbation. The resulting robust parameters Θr can be written as

(cid:16)

(cid:17)

µ, Σ) with the NLL under adversarial

Θr = arg min

µ,Σ

(x,y)

max

(cid:107)2≤

(cid:107)

∼D (cid:20)

(cid:96)(x + δ; y

µ, Σ)

(cid:21)

(12)

A detailed analysis of this setting is in Appendix E—here we present a high-level overview of the results.

(1) Vulnerability from metric misalignment (non-robust features). Note that in this model, one can rig-

orously make reference to an inner product (and thus a metric) induced by the features. In particular, one

can view the learned parameters of a Gaussian Θ = (µ, Σ) as deﬁning an inner product over the input space

µ). This in turn induces the Mahalanobis distance, which represents

given by

how a change in the input affects the features learned by the classiﬁer. This metric is not necessarily aligned

with the metric in which the adversary is constrained, the (cid:96)2-norm. Actually, we show that adversarial

vulnerability arises exactly as a misalignment of these two metrics.

Θ = (x

µ)(cid:62)

1(y

x, y

−

(cid:104)

(cid:105)

Theorem 1 (Adversarial vulnerability from misalignment). Consider an adversary whose perturbation is deter-

mined by the “Lagrangian penalty” form of (12), i.e.

(cid:96)(x + δ; y

max

µ, Σ)

(cid:107)2,

· (cid:107)

−

) is a constant trading off NLL minimization and the adversarial constraint14. Then, the adversarial

where C

loss

≥

σmin(Σ

∗

Ladv incurred by the non-robustly learned (µ, Σ) is given by:

Ladv(Θ)

I + (C

(Θ) = tr

− L

and, for a ﬁxed tr(Σ

∗

) = k the above is minimized by Σ

(cid:20)(cid:16)

∗

= k

d I.

∗ −

I)−

(cid:21)

(cid:17)

−

In fact, note that such a misalignment corresponds precisely to the existence of non-robust features, as it

indicates that “small” changes in the adversary’s metric along certain directions can cause large changes

under the data-dependent notion of distance established by the parameters. This is illustrated in Figure 4,

where misalignment in the feature-induced metric is responsible for the presence of a non-robust feature in

the corresponding classiﬁcation problem.

(2) Robust Learning. The optimal (non-robust) maximum likelihood estimate is Θ = Θ

∗, and thus the

vulnerability for the standard MLE estimate is governed entirely by the true data distribution. The follow-

ing theorem characterizes the behaviour of the learned parameters in the robust problem. 15. In fact, we can

prove (Section E.3.4) that performing (sub)gradient descent on the inner maximization (also known as ad-

versarial training [GSS15; Mad+18]) yields exactly Θr. We ﬁnd that as the perturbation budget ε is increased,

the metric induced by the learned features mixes (cid:96)2 and the metric induced by the features.

Theorem 2 (Robustly Learned Parameters). Just as in the non-robust case, µr = µ∗, i.e. the true mean is learned.

For the robust covariance Σr, there exists an ε0 > 0, such that for any ε

[0, ε0),

Σr =

∗

λ ·

I +

λ ·

∗

Σ2

∗

(cid:114)

where

(cid:32)

∈

1 + ε1/2

ε1/2 + ε3/2 (cid:33) ≤

≤

1 + ε1/2

(cid:32)

ε1/2 (cid:33)

The effect of robust optimization under an (cid:96)2-constrained adversary is visualized in Figure 4. As (cid:101)

grows, the learned covariance becomes more aligned with identity. For instance, we can see that the classi-

ﬁer learns to be less sensitive in certain directions, despite their usefulness for natural classiﬁcation.

(3) Gradient Interpretability.

Tsipras et al. [Tsi+19] observe that gradients of robust models tend to look

more semantically meaningful. It turns out that under our model, this behaviour arises as a natural con-

sequence of Theorem 2. In particular, we show that the resulting robustly learned parameters cause the

gradient of the linear classiﬁer and the vector connecting the means of the two distributions to better align

(in a worst-case sense) under the (cid:96)2 inner product.

Theorem 3 (Gradient alignment). Let f (x) and fr(x) be monotonic classiﬁers based on the linear separator induced

by standard and (cid:96)2-robust maximum likelihood classiﬁcation, respectively. The maximum angle formed between the

gradient of the classiﬁer (wrt input) and the vector connecting the classes can be smaller for the robust model:

∇x fr(x)

µ,

(cid:104)

(cid:105)

(...