On Evaluating Adversarial Robustness

advertisement as natural content (Tram r et al., 2018), or cause a digital assistant

to incorrectly recognize commands it is given (Carlini et al., 2016).

" To test the worst-case robustness of machine learning algorithms. Many

real-world environments have inherent randomness that is dicult to predict. By

analyzing the robustness of a model from the perspective of an adversary, we can

estimate the worst-case robustness in a real-world setting. Through random testing,

it can be dicult to distinguish a system that fails one time in a billion from a

system that never fails: even when evaluating such a system on a million choices of

randomness, there is just under 0.1% chance to detect a failure case.

However, analyzing the worst-case robustness can discover a dierence. If a powerful

adversary who is intentionally trying to cause a system to misbehave (according to

some denition) cannot succeed, then we have strong evidence that the system will

not misbehave due to any unforeseen randomness.

" To measure progress of machine learning algorithms towards human-level

abilities. To advance machine learning algorithms it is important to understand

where they fail. In terms of performance, the gap between humans and machines

is quite small on many widely studied problem domains, including reinforcement

learning (e.g., Go and Chess (Silver et al., 2016)) or natural image classication

(Krizhevsky et al., 2012). In terms of adversarial robustness, however, the gap be-

tween humans and machines is astonishingly large: even in settings where machine

learning achieves super-human accuracy, an adversary can often introduce pertur-

bations that reduce their accuracy to levels of random guessing and far below the

accuracy of even the most uninformed human.3 This suggests a fundamental dif-

ference of the decision-making process of humans and machines. From this point

of view, adversarial robustness is a measure of progress in machine learning that is

orthogonal to performance.

The motivation for why the research was conducted informs the methodology through which

it should be evaluated: a paper that sets out to prevent a real-world adversary from fooling

a specic spam detector assuming the adversary can not directly access the underlying

model will have a very dierent evaluation than one that sets out to measure the worst-case

robustness of a self-driving cars vision system.

2While this threat model is often repeated in the literature, it may have limited impact for

real-world adversaries, who in practice may have have little nancial motivation to cause harm to

self-driving cars.

3Note that time-limited humans appear vulnerable to some forms of adversarial exam-

ples (Elsayed et al., 2018).

3

On Evaluating Adversarial Robustness

This paper therefore does not (and could not) set out to provide a denitive answer for how

all evaluations should be performed. Rather, we discuss methodology that we believe is

common to most evaluations. Whenever we provide recommendations that may not apply

to some class of evaluations, we state this fact explicitly. Similarly, for advice we believe

holds true universally, we discuss why this is the case, especially when it may not be obvious

at rst.

The remainder of this section provides an overview of the basic methodology for a defense

evaluation.

2.2 Threat Models

A threat model species the conditions under which a defense is designed to be secure and

the precise security guarantees provided; it is an integral component of the defense itself.

Why is it important to have a threat model? In the context of a defense where the purpose

is motivated by security, the threat model outlines what type of actual attacker the defense

intends to defend against, guiding the evaluation of the defense.

However, even in the context of a defense motivated by reasons beyond security, a threat

model is necessary for evaluating the performance of the defense. One of the dening

properties of scientic research is that it is falsiable: there must exist an experiment that

can contradict its claims. Without a threat model, defense proposals are often either not

falsiable or trivially falsiable.

Typically, a threat model includes a set of assumptions about the adversarys goals, knowl-

edge, and capabilities. Next, we briey describe each.

2.2.1 Adversary goals

How should we dene an adversarial example? At a high level, adversarial examples can

be dened as inputs specically designed to force a machine learning system to produce

erroneous outputs. However, the precise goal of an adversary can vary signicantly across

dierent settings.

For example, in some cases the adversarys goal may be to simply cause misclassication

any input being misclassied represents a successful attack. Alternatively, the adversary

may be interested in having the model misclassify certain examples from a source class into

a target class of their choice. This has been referred to a source-target misclassication

attack (Papernot et al., 2016b) or targeted attack (Carlini & Wagner, 2017c).

In other settings, only specic types of misclassication may be interesting. In the space

of malware detection, defenders may only care about the specic source-target class pair

where an adversary causes a malicious program to be misclassied as benign; causing a

benign program to be misclassied as malware may be uninteresting.

2.2.2 Adversarial capabilities

In order to build meaningful defenses, we need to impose reasonable constraints to the

attacker. An unconstrained attacker who wished to cause harm may, for example, cause

bit-ips on the weights of the neural network, cause errors in the data processing pipeline,

backdoor the machine learning model, or (perhaps more relevant) introduce large perturba-

tions to an image that would alter its semantics. Since such attacks are outside the scope of

defenses adversarial examples, restricting the adversary is necessary for designing defenses

that are not trivially bypassed by unconstrained adversaries.

To date, most defenses to adversarial examples typically restrict the adversary to making

small changes to inputs from the data-generating distribution (e.g.

inputs from the test

set). Formally, for some natural input x and similarity metric D, x2 is considered a valid

4

On Evaluating Adversarial Robustness

adversarial example if D(x, x2) d for some small and x2 is misclassied4. This denition

is motivated by the assumption that small changes under the metric D do not change the

true class of the input and thus should not cause the classier to predict an erroneous class.

A common choice for D, especially for the case of image classication, is dening it as

the p-norm between two inputs for some p.

(For instance, an -norm constraint of

for image classication implies that the adversary cannot modify any individual pixel by

more than .) However, a suitable choice of D and may vary signicantly based on the

particular task. For example, for a task with binary features one may wish to study 0-

bounded adversarial examples more closely than -bounded ones. Moreover, restricting

adversarial perturbations to be small may not always be important: in the case of malware

detection, what is required is that the adversarial program preserves the malware behavior

while evading ML detection.

Nevertheless, such a rigorous and precise denition of the adversarys capability, leads to

well-dened measures of adversarial robustness that are, in principle, computable. For

example, given a model f ( ), one common way to dene robustness is the worst-case loss L

for a given perturbation budget,

E

(x,y)

max

x2:D(x,x2)<

L

f (x2), y

(cid:0)

(cid:21)

(cid:1)

.

Another commonly adopted denition is the average (or median) minimum-distance of the

adversarial perturbation,

min

x2Ax,y

where Ax,y depends on the denition of adversarial example, e.g. Ax,y = {x2 | f (x2) 6= y}

for misclassication or Ax,y = {x | f (x2) = t} for some target class t.

E

(x,y)

D(x, x2)

(cid:21)

,

A key challenge of security evaluations is that while this adversarial risk (Madry et al., 2017;

Uesato et al., 2018) is often computable in theory (e.g. with optimal attacks or brute force

enumeration of the considered perturbations), it is usually intractable to compute exactly,

and therefore in practice we must approximate this quantity. This diculty is at the heart of

why evaluating worst-case robustness is dicult: while evaluating average-case robustness

is often as simple as sampling a few hundred (or thousand) times from the distribution and

computing the mean, such an approach is not possible for worst-case robustness.

Finally, a common, often implicit, assumption in adversarial example research is that the

adversary has direct access to the models input features: e.g., in the image domain, the

adversary directly manipulates the image pixels. However, in certain domains, such as

malware detection or language modeling, these features can be dicult to reverse-engineer.

As a result, dierent assumptions on the capabilities of the adversary can signicantly

impact the evaluation of a defenses eectiveness.

Comment on p-norm-constrained threat models. A large body of work studies a

threat model where the adversary is constrained to p-bounded perturbations. This threat

model is highly limited and does not perfectly match real-world threats (Engstrom et al.,

2017; Gilmer et al., 2018). However, the well-dened nature of this threat model is helpful

for performing principled work towards building strong defenses. While p-robustness does

not imply robustness in more realistic threat models, it is almost certainly the case that lack

of robustness against p-bounded perturbation will imply lack of robustness in more realistic

threat models. Thus, working towards solving robustness for these well-dened p-bounded

threat models is a useful exercise.

2.2.3 Adversary knowledge.

A threat model clearly describes what knowledge the adversary is assumed to have. Typi-

cally, works assume either white-box (complete knowledge of the model and its parameters)

4 It is often required that the original input x is classied correctly, but this requirement can vary

an adversarial example as long as it is classied dierently

across papers. Some papers consider x2

from x.

5

On Evaluating Adversarial Robustness

or black-box access (no knowledge of the model) with varying degrees of black-box access

(e.g., a limited number of queries to the model, access to the predicted probabilities or just

the predicted class, or access to the training data).

In general, the guiding principle of a defenses threat model is to assume that the adversary

has complete knowledge of the inner workings of the defense. It is not reasonable to assume

the defense algorithm can be held secret, even in black-box threat models. This widely-

held principle is known in the eld of security as Kerckhos principle (Kerckhos, 1883),

and the opposite is known as security through obscurity. The open design of security

mechanisms is a cornerstone of the eld of cryptography (Saltzer & Schroeder, 1975). This

paper discusses only how to perform white-box evaluations, which implies robustness to

black-box adversaries, but not the other way around.

Holding Data Secret. While it can be acceptable to hold some limited amount of in-

formation secret, the dening characteristic of a white-box evaluation (as we discuss in this

paper) is that the threat model assumes the attacker has full knowledge of the underlying

system.

That does not mean that all information has to be available to the adversaryit can be

acceptable for the defender to hold a small amount of information secret. The eld of

cryptography, for example, is built around the idea that one can keep secret the encryption

keys, but the underlying algorithm is be assumed to be public.

A defense that holds values secret should justify that it is reasonable to do so. In particular,

secret information generally satises at least the following two properties:

1. The secret must be easily replaceable. That is, there should be an ecient algorithm

to generate a new secret if the prior one happened to be leaked.

2. The secret must be nonextractable. An adversary who is allowed to query the system

should not be able to extract any information about the secret.

For example, a defense that includes randomness (chosen fresh) at inference time is using

secret information not available to the adversary. As long as the distribution is known, this

follows Kerckhos principle. On the other hand, if a single xed random vector was added

to the output of the neural network after classifying an input, this would not be a good

candidate for a secret. By subtracting the observed output of the model with the expected

output, the secret can be easily determined.

2.3 Restrict Attacks to the Defenses Threat Model

Attack work should always evaluate defenses under the threat model the defense states. For

example, if a defense paper explicitly states we intend to be robust to L2 attacks of norm

no greater than 1.5, an attack paper must restrict its demonstration of vulnerabilities in

the defense to the generation of adversarial examples with L2 norm less than 1.5. Showing

something dierent, e.g., adversarial examples with L norm less than 0.1, is important

and useful research5 (because it teaches the research community something that was not

previously known, namely, that this system may have limited utility in practice), but is not

a break of the defense: the defense never claimed to be robust to this type of attack.

2.4 Skepticism of Results

When performing scientic research one must be skeptical of all results. As Feynman con-

cisely put it, the rst principle is that you must not fool yourselfand you

are the easiest person to fool. This is never more true than when considering security

evaluations. After spending signicant eort to try and develop a defense that is robust

against attacks, it is easy to assume that the defense is indeed robust, especially when base-

line attacks fail to break the defense. However, at this time the authors need to completely

5See for example the work of Sharma & Chen (2017); Song et al. (2018) who explicitly step

outside of the threat model of the original defenses to evaluate their robustness.

6

On Evaluating Adversarial Robustness

switch their frame of mind and try as hard as possible to show their proposed defense is

ineective.6

Adversarial robustness is a negative goal for a defense to be truly eective, one needs to

show that no attack can bypass it. It is only that by failing to show the defense is ineective

to adaptive attacks (see below) that we can believe it will withstand future attack by a

motivated adversary (or, depending on the motivation of the research, that the claimed

lower bound is in fact an actual lower bound).

2.5 Adaptive Adversaries

After a specic threat model has been dened, the remainder of the evaluation focuses on

adaptive adversaries 7 which are adapted to the specic details of the defense and attempt

to invalidate the robustness claims that are made.

This evaluation is the most important section of any paper that develops a defense. After

the defense has been dened, ask: what attack could possibly defeat this defense? All attacks

that might work must be shown to be ineective. An evaluation that does not attempt to

do this is fundamentally awed.

Just applying existing adversarial attacks with default hyperparameters is not sucient,

even if these attacks are state-of-the-art: all existing attacks and hyperparameters have

been adapted to and tested only against existing defenses, and there is a good chance these

attacks will work sub-optimally or even fail against a new defense. A typical example

is gradient masking (Tram r et al., 2017), in which defenses manipulate the models gra-

dients and thus prevent gradient-based attacks from succeeding. However, an adversary

aware of the defense may recover these gradients through a black-box input-label queries,

as shown by Papernot et al. (2017), or through a dierent loss function, as demonstrated

by Athalye et al. (2018). In other words, gradient masking may make optimization-based

attacks fail but that does not mean that the space of adversarial perturbations decreased.

Defending against non-adaptive attacks is necessary but not sucient. It is our rm belief

that an evaluation against non-adaptive attacks is of very limited utility.

Along the same lines, there is no justication to study a zero-knowledge (Biggio et al.,

2013) threat model where the attacker is not aware of the defense. Defending against such

an adversary is an absolute bare-minimum that in no way suggests a defense will be eective

to further attacks. Carlini & Wagner (2017a) considered this scenario only to demonstrate

that some defenses were completely ineective even against this very weak threat model.

The authors of that work now regret not making this explicit and discourage future work

from citing this paper in support of the zero-knowledge threat model.

It is crucial to actively attempt to defeat the specic defense being proposed. On the most

fundamental level this should include a range of suciently dierent attacks with carefully

tuned hyperparameters. But the analysis should go deeper than that: ask why the defense

might prevent existing attacks from working optimally and how to customize existing attacks

or how to design completely new adversarial attacks to perform as well as possible. That

is, applying the same mindset that a future adversary would apply is the only way to show

that a defense might be able to withstand the test of time.

These arguments apply independent of the specic motivation of the robustness evaluation:

security, worst-case bounds or human-machine gap all need a sense of the maximum vulner-

ability of a given defense. In all scenarios we should assume the existence of an innitely

thorough adversary who will spend whatever time is necessary to develop the optimal

attack.

6One of the reasons it is so easy to accidentally fool oneself in security is that mistakes are very

dicult to catch. Very often attacks only fail because of a (correctable) error in how they are being

applied. It has to be the objective of the defense researcher to ensure that, when attacks fail, it is

because the defense is correct, and not because of an error in applying the attacks.

7We use the word adaptive adversary (and adaptive attack) to refer to the general notion

in security of an adversary (or attack, respectively) that adapts to what the defender has done

(Herley & van Oorschot, 2017; Carlini & Wagner, 2017a).

7

On Evaluating Adversarial Robustness

2.6 Reproducible Research: Code & Pre-trained Models

Even the most carefully-performed robustness evaluations can have subtle but fundamental

aws. We strongly believe that releasing full source code and pre-trained models is one of

the most useful methods for ensuring the eventual correctness of an evaluation. Releasing

source code makes it much more likely that others will be able to perform their own analysis

of the defense.8 Furthermore, completely specifying all defense details in a paper can be

dicult, especially in the typical 8-page limit of many conference papers. The source code

for a defense can be seen as the denitive reference for the algorithm.

It is equally important to release pre-trained models, especially when the resources that

would be required to train a model would be prohibitive to some researchers with limited

compute resources. The code and model that is released should be the model that was used

to perform the evaluation in the paper to the extent permitted by underlying frameworks for

accelerating numerical computations performed in machine learning. Releasing a dierent

model than was used in the paper makes it signicantly less useful, as any comparisons

against the paper may not be identical.

Finally, it is helpful if the released code contains a simple one-line script which will run the

full defense end-to-end on the given input. Note that this is often dierent than what the

defense developers want, who often care most about performing the evaluation as eciently

as possible. In contrast, when getting started with evaluating a defense (or to conrm any

results), it is often most useful to have a simple and correct method for running the full

defense over an input.

There are several

frameworks such as CleverHans (Papernot et al., 2018) or Fool-

box (Rauber et al., 2017) as well as websites9,10,11 which have been developed to assist

in this process.

3 Specific Recommendations: Evaluation Checklist

While the above overview is general-purpose advice we believe will stand the test of time,

it can be dicult to extract specic, actionable items from it. To help researchers today

perform more thorough evaluations, we now develop a checklist that lists common evaluation

pitfalls when evaluating adversarial robustness. Items in this list are sorted (roughly) into

three categories.

The items contained below are neither necessary nor sucient for performing a complete

adversarial example evaluation, and are intended to list common evaluation aws. There

likely exist completely ineective defenses which satisfy all of the below recommendations;

conversely, some of the strongest defenses known today do not check o all the boxes below

(e.g. Madry et al. (2017)).

We encourage readers to be extremely careful and not directly follow this list to perform

an evaluation or decide if an evaluation that has been performed is sucient. Rather, this

list contains common aws that are worth checking for to identify potential evaluation aws.

Blindly following the checklist without careful thought will likely be counterproductive: each

item in the list must be taken into consideration within the context of the specic defense

being evaluated. Each item on the list below is present because we are aware of several

defense evaluations which were broken and following that specic recommendation would

have revealed the aw. We hope this list will be taken as a collection of recommendations

that may or may not apply to a particular defense, but have been useful in the past.

This checklist is a living document that lists the most common evaluation aws as of

18 February, 2019. We expect the evaluation aws that are common today will not be

8In their analysis of the ICLR 2018 defenses (Athalye et al., 2018), the authors spent ve

times longer re-implementing the defenses than performing the security evaluation of the re-

implementations.

9https://robust-ml.org

10https://robust.vision/benchmark/leaderboard/

11https://foolbox.readthedocs.io/en/latest/modules/zoo.html

8

On Evaluating Adversarial Robustness

the most common aws in the future. We intend to keep this checklist up-to-date with

the latest recommendations for evaluating defenses by periodically updating its contents.

Readers should check the following URL for the most recent revision of the checklist:

https://github.com/evaluating-adversarial-robustness/adv-eval-paper.

3.1 Common Severe Flaws

There are several common severe evaluation aws which have the potential to completely in-

validate any robustness claims. Any evaluation which contains errors on any of the following

items is likely to have fundamental and irredeemable aws. Evaluations which intentionally

deviate from the advice here may wish to justify the decision to do so.

" 3 Do not mindlessly follow this list; make sure to still think about the evaluation.

" 2.2 State a precise threat model that the defense is supposed to be eective under.

The threat model assumes the attacker knows how the defense works.

The threat model states attackers goals, knowledge and capabilities.

For security-justied defenses, the threat model realistically models some adversary.

For worst-case randomized defenses, the threat model captures the perturbation space.

Think carefully and justify any p bounds placed on the adversary.

" 2.5 Perform adaptive attacks to give an upper bound of robustness.

The attacks are given access to the full defense, end-to-end.

The loss function is changed as appropriate to cause misclassication.

4.3 Focus on the strongest attacks for the threat model and defense considered.

" 2.6 Release pre-trained models and source code.

Include a clear installation guide, including all dependencies.

There is a one-line script which will classify an input example with the defense.

" 4.2 Report clean model accuracy when not under attack.

For defenses that abstain or reject inputs, generate a ROC curve.

" 5.2 Perform basic sanity tests on attack success rates.

Verify iterative attacks perform better than single-step attacks.

Verify increasing the perturbation budget strictly increases attack success rate.

With high distortion, model accuracy should reach levels of random guessing.

" 5.3 Generate an attack success rate vs. perturbation budget curve.

Verify the x-axis extends so that attacks eventually reach 100% success.

For unbounded attacks, report distortion and not success rate.

" 5.4 Verify adaptive attacks perform better than any other.

Compare success rate on a per-example basis, rather than averaged across the dataset.

Evaluate against some combination of black-box, transfer, and random-noise attacks.

" 5.7 Describe the attacks applied, including all hyperparameters.

3.2 Common Pitfalls

There are other common pitfalls that may prevent the detection of ineective defenses.

This list contains some potential pitfalls which do not apply to large categories of defenses.

However, if applicable, the items below are still important to carefully check they have been

applied correctly.

" 4.3 Apply a diverse set of attacks (especially when training on one attack approach).

Do not blindly apply multiple (nearly-identical) attack approaches.

" 4.4 Try at least one gradient-free attack and one hard-label attack.

Try Chen et al. (2017b); Uesato et al. (2018); Ilyas et al. (2018a); Brendel et al. (2017).

9

On Evaluating Adversarial Robustness

Check that the gradient-free attacks succeed less often than gradient-based attacks.

Carefully investigate attack hyperparameters that aect success rate.

" 4.5 Perform a transferability attack using a similar substitute model.

Select a substitute model as similar to the defended model as possible.

Generate adversarial examples that are initially assigned high condence.

Check that the transfer attack succeeds less often than white-box attacks.

" 4.6 For randomized defenses, properly ensemble over randomness.

Verify that attacks succeed if randomness is assigned to one xed value.

State any assumptions about adversary knowledge of randomness in the threat model.

" 4.7 For non-dierentiable components, apply dierentiable techniques.

Discuss why non-dierentiable components were necessary.

Verify attacks succeed on undefended model with those non-dierentiable components.

Consider applying BPDA (Athalye et al., 2018) if applicable.

" 4.8 Verify that the attacks have converged under the selected hyperparameters.

Verify that doubling the number of iterations does not increase attack success rate.

Plot attack eectiveness versus the number of iterations.

Explore dierent choices of the step size or other attack hyperparameters.

" 4.9 Carefully investigate attack hyperparameters and report those selected.

Start search for adversarial examples at a random oset.

Investigate if attack results are sensitive to any other hyperparameters.

" 5.1 Compare against prior work and explain important dierences.

When contradicting prior work, clearly explain why dierences occur.

Attempt attacks that are similar to those that defeated previous similar defenses.

When comparing against prior work, ensure it has not been broken.

" 4.10 Test broader threat models when proposing general defenses. For images:

Apply rotations and translations (Engstrom et al., 2017).

Apply common corruptions and perturbations (Hendrycks & Dietterich, 2018).

Add Gaussian noise of increasingly large standard deviation (Ford et al., 2019).

3.3 Special-Case Pitfalls

The following items apply to a smaller fraction of evaluations. Items presented here are

included because while they may diagnose aws in some defense evaluations, they are not

necessary for many others. In other cases, the tests presented here help provide additional

evidence that the evaluation was performed correctly.

" 4.1 Investigate if it is possible to use provable approaches.

Examine if the model is amenable to provable robustness lower-bounds.

" 4.11 Attack with random noise of the correct norm.

For each example, try 10,000+ dierent choices of random noise.

Check that the random attacks succeed less-often than white-box attacks.

" 4.12 Use both targeted and untargeted attacks during evaluation.

State explicitly which attack type is being used.

" 4.13 Perform ablation studies with combinations of defense components removed.

Attack a similar-but-undefended model and verify attacks succeed.

If combining multiple defense techniques, argue why they combine usefully.

" 4.14 Validate any new attacks by attacking other defenses.

Attack other defenses known to be broken and verify the attack succeeds.

10

On Evaluating Adversarial Robustness

Construct synthetic intentionally-broken models and verify the attack succeeds.

Release source code for any new attacks implemented.

" 5.5 Investigate applying the defense to domains other than images.

State explicitly if the defense applies only to images (or another domain).

f (a(x)), not min

aA

" 5.6 Report per-example attack success rate: mean

xX

min

aA

mean

xX

f (a(x)).

4 Evaluation Recommendations

We now expand on the above checklist and provide the rationale for each item.

4.1 Investigate Provable Approaches

With the exception of this subsection, all other advice in this paper focuses on performing

heuristic robustness evaluations. Provable robustness approaches are preferable to only

heuristic ones. Current provable approaches often can only be applied when the neural

network is explicitly designed with the objective of making these specic provable techniques

applicable (Kolter & Wong, 2017; Raghunathan et al., 2018; Weng et al., 2018). While this

approach of designing-for-provability has seen excellent progressthe best approaches today

can certify some (small) robustness even on ImageNet classiers (Lecuyer et al., 2018)

often the best heuristic defenses oer orders of magnitude better (estimated) robustness.

Proving a lower bound of defense robustness guarantees that the robustness will n...