Adversarial Examples and Adversarial Training

Maxout

Carefully tuned sigmoid

Carefully tuned sigmoid

LSTM

LSTM

Google Proprietary

(Goodfellow 2016)

Nearly Linear Responses in Practice

x

a

m

t

f

o

s

o

t

t

n

e

m

u

g

r

A

(Goodfellow 2016)

Small inter-class distances

Clean

example

Perturbation Corrupted

example

Perturbation changes the true

class

Random perturbation does not

change the class

Perturbation changes the input

to “rubbish class”

All three perturbations have L2 norm 3.96

This is actually small. We typically use 7!

(Goodfellow 2016)

The Fast Gradient Sign Method

(Goodfellow 2016)

Maps of Adversarial and Random

Cross-Sections

(collaboration with David Warde-Farley and Nicolas Papernot)

(Goodfellow 2016)

Maps of Adversarial Cross-Sections

(Goodfellow 2016)

Maps of Random Cross-Sections

Adversarial examples

are not noise

(collaboration with David Warde-Farley and Nicolas Papernot)

(Goodfellow 2016)

Estimating the Subspace

Dimensionality

(Tramèr et al, 2017)

(Goodfellow 2016)

Clever Hans

(“Clever Hans,

Clever

Algorithms,”

Bob Sturm)

(Goodfellow 2016)

Wrong almost everywhere

(Goodfellow 2016)

Adversarial Examples for RL

(Huang et al., 2017)

(Goodfellow 2016)

High-Dimensional Linear Models

Weights

Clean examples

Adversarial

Signs of weights

(Goodfellow 2016)

Linear Models of ImageNet

(Andrej Karpathy, “Breaking Linear Classifiers on ImageNet”)

(Goodfellow 2016)

RBFs behave more intuitively

(Goodfellow 2016)

Cross-model, cross-dataset

generalization

(Goodfellow 2016)

Cross-technique transferability

(Papernot 2016)

(Goodfellow 2016)

Transferability Attack

Target model with

unknown weights,

machine learning

algorithm, training

set; maybe non-

differentiable

Train your

own model

Substitute model

mimicking target

model with known,

differentiable function

Deploy adversarial

examples against the

target; transferability

property results in them

succeeding

Adversarial

examples

Adversarial crafting

against substitute

(Goodfellow 2016)

Cross-Training Data Transferability

Strong

Weak

Intermediate

(Papernot 2016)

(Goodfellow 2016)

Enhancing Transfer With

Ensembles

(Liu et al, 2016)

(Goodfellow 2016)

Adversarial Examples in the

Human Brain

These are

concentric

circles,

not

intertwined

spirals.

(Pinna and Gregory, 2002)

(Goodfellow 2016)

Practical Attacks

• Fool real classifiers trained by remotely hosted API

(MetaMind, Amazon, Google)

• Fool malware detector networks

• Display adversarial examples in the physical world

and fool machine learning systems that perceive

them through a camera

(Goodfellow 2016)

Adversarial Examples in the

Physical World

(Kurakin et al, 2016)

(Goodfellow 2016)

Failed defenses

Generative

pretraining

Removing perturbation

with an autoencoder

Adding noise

at test time

Ensembles

Confidence-reducing

perturbation at test time

Error correcting

codes

Multiple glimpses

Weight decay

Various

non-linear units

Double backprop

Dropout

Adding noise

at train time

(Goodfellow 2016)

Generative Modeling is not

Sufficient to Solve the Problem

(Goodfellow 2016)

Universal approximator theorem

Universal approximator

theorem

Neural nets can represent either function:

Neural nets can represent either function:

Maximum likelihood doesn’t cause them to

Maximum likelihood doesn’t cause them to learn

learn the right function. But we can fix that...

the right function. But we can fix that...

(Goodfellow 2016)

Google Proprietary

Training on Adversarial Examples

100

e

t

a

r

n

o

i

t

a

c

fi

i

s

s

a

l

c

s

i

m

t

s

e

T

1

10

2

10

Train=Clean, Test=Clean

Train=Clean, Test=Adv

Train=Adv, Test=Clean

Train=Adv, Test=Adv

0

50

100

150

200

250

300

Training time (epochs)

(Goodfellow 2016)

Adversarial Training of other

Models

• Linear models: SVM / linear regression cannot learn

a step function, so adversarial training is less useful,

very similar to weight decay

• k-NN: adversarial training is prone to overfitting.

• Takeway: neural nets can actually become more

secure than other models. Adversarially trained

neural nets have the best empirical success rate on

adversarial examples of any machine learning model.

(Goodfellow 2016)

Weaknesses Persist

(Goodfellow 2016)

Adversarial Training

Labeled as bird

Still has same label (bird)

Decrease

probability

of bird class

(Goodfellow 2016)

Virtual Adversarial Training

New guess should

Unlabeled; model

match old guess

guesses it’s probably

(probably bird, maybe plane)

a bird, maybe a plane

Adversarial

perturbation

intended to

change the guess

(Goodfellow 2016)

Text Classification with VAT

RCV1 Misclassification Rate

8.00

7.50

7.00

6.50

6.00

Earlier SOTA

SOTA

Our baseline Adversarial

Virtual

Adversarial

Both

Both +

bidirectional model

Zoomed in for legibility

(Goodfellow 2016)

6.686.977.057.127.407.207.70Universal engineering machine (model-based optimization)

Make new inventions

by finding input

that maximizes

model’s predicted

performance

Training data Extrapolation

(Goodfellow 2016)

Conclusion

• Attacking is easy

• Defending is difficult

• Adversarial training provides regularization and

semi-supervised learning

• The out-of-domain input problem is a bottleneck for

model-based optimization generally

(Goodfellow 2016)

cleverhans

Open-source library available at:

https://github.com/openai/cleverhans

Built on top of TensorFlow (Theano support anticipated)

Standard implementation of attacks, for adversarial training

and reproducible benchmarks

(Goodfellow 2016)