Adversarial Attacks and Defences: A Survey

Papernot et al. [57] addressed the aspect of generalization of adversarial examples. Abadi et al. [8]

introduced the concept of distributed deep learning as a way to protect the privacy of training

data. Recently in 2017, Hitaj et al. [34] exploited the real-time nature of the learning models to

train a Generative Adversarial Network and showed that the privacy of the collaborative systems

can be jeopardised. Since the findings of Szegedy, a lot of attention has been drawn to the context

of adversarial learning and the security of deep neural networks. A number of countermeasures

have been proposed in recent years to mitigate the effects of adversarial attacks. Kurakin et al.

[43] came up with the idea of using adversarial training to protect the learner by augmenting

the training set using both original and perturbed data. Hinton et al. [33] introduced the concept

of distillation which was used by Papernot et al. [62] to propose a defensive mechanism against

adversarial examples. Samangouei et al. [63] proposed a mechanism to use Generative Adversarial

Network as a countermeasure for adversarial perturbations. Although each of these proposed

defense mechanisms were found to be efficient against particular classes of attacks, none of them

could be used as a one-stop solution for all kinds of attacks. Moreover, implementation of these

defense strategies can lead to degradation of performance and efficiency of the concerned model.

1.1 Motivation and Contribution

The importance of Deep learning applications is increasing day-by-day in our daily life. However,

these deep learning applications are vulnerable to adversarial attacks. To the best of our knowledge,

there has been a little exhaustive survey in the field of adversarial learning covering different types

of adversarial attacks and their countermeasures. Akhtar et al. [9] presented a comprehensive

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

Adversarial Attacks and Defences: A Survey

x:3

survey on adversarial attacks on deep learning but in a restrictive context of computer vision.

There have been a handful of surveys on security evaluation related to particular machine learning

applications [13], [12], [23], [17]. Kumar et al. [42] provided a comprehensive survey of prior works

by categorizing the attacks under four overlapping classes. The primary motivation of this paper is

to summarize recent advances in different types of adversarial attacks with their countermeasures

by analyzing various threat models and attack scenarios. We follow a similar approach like prior

surveys but without restricting ourselves to specific applications and also in a more elaborate

manner with practical examples.

Organization

In this paper, we review recent findings on adversarial attacks and present a detailed understanding

of the attack models and methodologies. While our major focus is on attacks and defenses on

deep neural networks, we have also presented attack scenarios on Support Vector Machines (SVM)

keeping in mind their extensive use in real-world applications. In Section 2, we provide a taxonomy

of the related terms and keywords and categorize the threat models. This section also explains

adversarial capabilities and illustrates potential attack strategies in training (e.g. poisoning attack)

and testing (e.g. evasion attack) phases. We discuss in brief the basic notion of black box and white

box attacks with relevant applications and further classify black box attack based on how much

information is available to the adversary about the system. Section 3 summarizes exploratory

attacks that aim to learn algorithms and models of machine learning systems under attack. Since

the attack strategies in evasion and poisoning attacks often overlap, we have combined the work

focusing on both of them in Section 4. In Section 5 we discuss some of the current defense strategies

and we conclude in Section 6.

2 TAXONOMY OF MACHINE LEARNING AND ADVERSARIAL MODEL

Before discussing in details about the attack models and their countermeasures, in this section we

will provide a qualitative taxonomy on different terms and key words related to adversarial attacks

and categorize the threat models.

2.1 Keywords and Definitions

In this section, we summarize predominantly used approaches with emphasis on neural networks

to solve machine learning problems and their respective application.

" Support Vector Machines Support vector machines (SVMs) are supervised learning models

capable of constructing a hyperplane or a set of hyperplanes in high-dimensional space, which

can be used for classification, regression or outliers detection. In other words, a SVM model

is a representation of data as points in space with objective of building a maximum-margin

hyperplane and splitting the training examples into classes, while maximizing the distance

between the split points.

" Neural Networks: Artificial Neural networks (ANNs) inspired by the biological neural

networks is based on a collection of perceptrons called neurons. Each neuron maps a set of

inputs to output using an activation function. The learning governs the weights and activation

function so as to be able to correctly determine the output. Weights in a multi-layered feed

forward are updated by the back-propagation algorithm. Neuron was first introduced by

McCulloch-Pitts, followed by Hebbs learning rule, eventually giving rise to multi-layer

feed-forward perceptron and backpropagation algorithm. ANNs deal with supervised (CNN,

DNN) and unsupervised network models (self organizing maps) and their learning rules. The

neural network models used ubiquitously are discussed below.

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

x:4

A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, D. Mukhopadhyay

(1) DNN: While single layer neural net or perceptron is a feature-engineering approach,

deep neural network (DNN) enables feature learning using raw data as input. Multiple

hidden layers and its interconnections extract the features from unprocessed input and

thus enhances the performance by finding latent structures in unlabeled, unstructured

data. A typical DNN architecture, graphically depicted in Figure. 1, consists of multiple

successive layers (at least 2 hidden layers) of neurons. Each processing layer can be viewed

as learning a different, more abstract representation of the original multidimensional input

distribution. As a whole, a DNN can be viewed as a highly complex function that is capable

of nonlinearly mapping original high-dimensional data points to a lower dimensional space.

Fig. 1. Deep Neural Network

(2) CNN: A Convolutional Neural Network (CNN) consists of one or more convolutional or

sub-sampling layers, followed by one or more fully connected layers, to share weights

and reduce the number of parameters. The architecture of CNN, shown in Figure. 2, is

designed to take advantage of 2D input structure (e.g. input image). Convolution layer

creates a feature map; pooling (also called sub-sampling or down-sampling) reduces the

dimensionality of each feature map but retains the most important informations to have a

model robust to small distortions. For example, to describe a large image, feature values in

original matrix can be aggregated at various locations (e.g. max-pooling) to form a matrix

of lower dimension. The last fully connected layer use the feature matrix formed from

previous layers to classify the data. CNN is mainly used for feature extraction, thus it also

finds application in data preprocessing commonly used in image recognition tasks.

2.2 Adversarial Threat Model

The security of any machine learning model is measured concerning the adversarial goals and

capabilities. In this section, we taxonomize the threat models in machine learning systems keeping

in mind the strength of the adversary. We begin with the identification of threat surface [61] of

systems built on machine learning models to identify where and how an adversary may attempt to

subvert the system under attack.

2.2.1 The Attack Surface. A system built on Machine Learning can be viewed as a generalized

data processing pipeline. A primitive sequence of operations of the system at the testing time can

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

Adversarial Attacks and Defences: A Survey

x:5

Fig. 2. Convolutional Neural Network for MNIST digit recognition

be viewed as: a) collection of input data from sensors or data repositories, b) transferring the data in

the digital domain, c) processing of the transformed data by machine learning model to produce an

output, and finally, d) action taken based on the output. For illustration, consider a generic pipeline

of an automated vehicle system as shown Figure 3.

Fig. 3. Generic pipeline of an Automated Vehicle System

The system collects sensor inputs (images using camera) from which model features (tensor of

pixel values) are extracted and used within the models. It then interprets the meaning of the output

(probability of stop sign), and takes appropriate action (stopping the car). The attack surface, in

this case, can be defined with respect to the data processing pipeline. An adversary can attempt

to manipulate either the collection or the processing of data to corrupt the target model, thus

tampering the original output. The main attack scenarios identified by the attack surface are

sketched below [17, 18]:

(1) Evasion Attack: This is the most common type of attack in the adversarial setting. The

adversary tries to evade the system by adjusting malicious samples during testing phase.

This setting does not assume any influence over the training data.

(2) Poisoning Attack: This type of attack, known as contamination of the training data, takes

place during the training time of the machine learning model. An adversary tries to poison

the training data by injecting carefully designed samples to compromise the whole learning

process eventually.

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

x:6

A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, D. Mukhopadhyay

(3) Exploratory Attack: These attacks do not influence training dataset. Given black box access

to the model, they try to gain as much knowledge as possible about the learning algorithm

of the underlying system and pattern in training data.

The definition of a threat model depends on the information the adversary has at their disposal.

Next, we discuss in details the adversarial capabilities for the threat model.

2.2.2 The Adversarial Capabilities. The term adversarial capabilities refer to the amount of

information available to an adversary about the system, which also indicates the attack vector he

may use on the threat surface. For illustration, again consider the case of an automated vehicle

system as shown in Figure 3 with the attack surface being the testing time (i.e., an Evasion Attack).

An internal adversary is one who have access to the model architecture and can use it to distinguish

between different images and traffic signs, whereas a weaker adversary is one who have access

only to the dump of images fed to the model during testing time. Though both the adversaries

are working on the same attack surface, the former adversary is assumed to have much more

information and is thus strictly stronger. We explore the range of adversarial capabilities in

machine learning systems as they relate to testing and training phases.

Training Phase Capabilities. Attacks during training time attempt to influence or corrupt the

model directly by altering the dataset used for training. The most straightforward and arguably the

weakest attack on training phase is by merely accessing a partial or full training data. There are

three broad attack strategies for altering the model based on the adversarial capabilities.

(1) Data Injection: The adversary does not have any access to the training data as well as to the

learning algorithm but has ability to augment a new data to the training set. He can corrupt

the target model by inserting adversarial samples into the training dataset.

(2) Data Modification: The adversary does not have access to the learning algorithm but has

full access to the training data. He poisons the training data directly by modifying the data

before it is used for training the target model.

(3) Logic Corruption: The adversary has the ability to meddle with the learning algorithm.

These attacks are referred as logic corruption. Apparently, it becomes very difficult to design

counter strategy against these adversaries who can alter the learning logic, thereby controlling

the model itself.

Testing Phase Capabilities. Adversarial attacks at the testing time do not tamper with the tar-

geted model but rather forces it to produce incorrect outputs. The effectiveness of such attacks

is determined mainly by the amount of information available to the adversary about the model.

Testing phase attacks can be broadly classified into either White-Box or Black-Box attacks. Before

discussing these attacks, we provide a formal definition of a training procedure for a machine

learning model.

Let us consider a target machine learning model f is trained over input pair (X , y) from the

data distribution with a randomized training procedure train having randomness r (e.g., random

weight initialization, dropout, etc.). The model parameters are learned after the training procedure.

More formally, we can write:

train(f , X , y, r )

Now, let us understand the capabilities of the white-box and black-box adversaries with respect

to this definition. An overview of the different threat models have been shown in Figure. 4

White-Box Attacks. In white-box attack on a machine learning model, an adversary has total

knowledge about the model (f ) used for classification (e.g., type of neural network along with

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

Adversarial Attacks and Defences: A Survey

x:7

Article

Black box

White box

Papernot, Nicolas[57]

Non-adaptive

Rosenberg, Ishai [65]

Adaptive

Tram r, Florian [73]

Non-Adaptive

Papernot, Nicolas [60]

(Non)-Adaptive

Fredrikson, Matt [26]

Adaptive

Shokri, Reza [67]

Adaptive

Hitaj, Briland [34]

Strict

Moosavi-Dezfooli [52]

Tram r, Florian [72]

Fig. 4. Overview of threat models in relevant articles

number of layers). The attacker has information about the algorithm (train) used in training (e.g.,

gradient-descent optimization) and can access the training data distribution ( ). He also knows the

parameters ( ) of the fully trained model architecture. The adversary utilizes available information

to identify the feature space where the model may be vulnerable, i.e, for which the model has

a high error rate. Then the model is exploited by altering an input using adversarial example

crafting method, which we discuss later. The access to internal model weights for a white-box

attack corresponds to a very strong adversarial attack.

Black-Box Attacks. Black-Box attack, on the contrary, assumes no knowledge about the model

and uses information about the settings or past inputs to analyse the vulnerability of the model.

For example, in an oracle attack, the adversary exploits a model by providing a series of carefully

crafted inputs and observing outputs. Black Box attacks can be further classified into the following

categories:

(1) Non-Adaptive Black-Box Attack: For a target model (f ), a non-adaptive black-box adver-

sary only gets access to the target models training data distribution ( ). The adversary then

chooses a procedure train2 for a model architecture f 2 and trains a local model over samples

from the data distribution to approximate the model learned by the target classifier. The

adversary crafts adversarial examples on the local model f 2 using white-box attack strategies

and applies these crafted inputs to the target model to force mis-classifications.

(2) Adaptive Black-Box Attack: For a target model (f ), an adaptive black-box adversary does

not have any information regarding the training process but can access the target model as an

oracle. This strategy is analogous to chosen-plaintext attack in cryptography. The adversary

issues adaptive oracle queries to the target model and labels a carefully selected dataset, i.e.,

for any arbitrarily chosen x the adversary obtains its label y by querying the target model f .

The adversary then chooses a procedure train2 and model architecture f 2 to train a surrogate

model over tuples (x, y) obtained from querying the target model. The surrogate model then

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

x:8

A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, D. Mukhopadhyay

produces adversarial samples by following white-box attack technique for forcing the target

model to mis-classify malicious data.

(3) Strict Black-Box Attack: A black-box adversary sometimes may not contain the data distri-

bution but has the ability to collect the input-output pairs (x, y) from the target classifier.

However, he can not change the inputs to observe the changes in output like an adaptive

attack procedure. This strategy is analogous to the known-plaintext attack in cryptography

and would most likely to be successful for a large set of input-output pairs.

The point to be remembered in the context of a black-box attack is that an adversary neither

tries to learn the randomness r used to train the target model nor the target models parameters .

The primary objective of a black-box adversary is to train a local model with the data distribution

in case of a non-adaptive attack and with carefully selected dataset by querying the target model

in case of an adaptive attack. Table 1 shows a brief distinction between black box and white box

attacks.

Description

Adversary

Knowledge

Attack

Strategy

Black box attack

Restricted knowledge from being

able to only observe the networks

output on some probed inputs.

Based on a greedy local search

generating an implicit approximation

to the actual gradient w.r.t

the current output by observing

changes in input.

White box attack

Detailed knowledge of the network

architecture and the parameters

resulting from training.

Based on the gradient of the network

loss function w.r.t to the input.

Table 1. Distinction between black box and white box attacks

The adversarial threat model also depends not only depends on the adversarial capabilities but

also on the action taken by the adversary. In the next subsection, we discuss the goal of an adversary

while compromising the security of any machine learning system.

2.2.3 Adversarial Goals. An adversary attempts to provide an input x to a classification system

that results in an incorrect output classification. The objective of the adversary is inferred from the

incorrectness of the model. Based on the impact on the classifier output integrity the adversarial

goals can be broadly classified as follows:

(1) Confidence Reduction: The adversary tries to reduce the confidence of prediction for the

target model. For example, a legitimate image of a stop sign can be predicted with a lower

confidence having a lesser probability of class belongingness.

(2) Misclassification: The adversary tries to alter the output classification of an input example

to any class different from the original class. For example, a legitimate image of a stop sign

will be predicted as any other class different from the class of stop sign.

(3) Targeted Misclassification: The adversary tries to produce inputs that force the output of

the classification model to be a specific target class. For example, any input image to the

classification model will be predicted as a class of images having go sign.

(4) Source/Target Misclassification: The adversary attempts to force the output of classifica-

tion for a specific input to be a particular target class. For example, the input image of stop

sign will be predicted as go sign by the classification model.

The taxonomy of the adversarial threat model for both the evasion and the poisoning attacks

with respect to the adversarial capabilities and adversarial goals are represented graphically in

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

Adversarial Attacks and Defences: A Survey

x:9

a) Attack Difficulty with respect to adversarial

capabilities and goals for Evasion Attacks

b) Attack Difficulty with respect to adversarial

capabilities and goals for Poisoning Attacks

Fig. 5. Taxonomy of Adversarial Model for a) Evasion Attacks and b) Poisoning Attacks with respect to

adversarial capabilities and goals

Figure 5. The horizontal axis of both figures represents the complexity of adversarial goals in

increasing order, and the vertical axis loosely represents the strength of an adversary in decreasing

order. The diagonal axis represents the complexity of a successful attack based on the adversarial

capabilities and goals.

Some of the noteworthy attacks along with their target applications is shown in Table. 2. Further

in Table. 3, we categorize those attacks under different threat models and discuss in detail about

them in the next section.

3 EXPLORATORY ATTACKS

Exploratory attacks do not modify the training set but instead tries to gain information about the

state by probing the learner. The adversarial examples are crafted in such a way that the learner

passes them as legitimate examples during testing phase.

3.1 Model Inversion(MI) Attack

Fredrikson et al. introduced "model inversion" in [27] where they considered a linear regression

model f that predicted drug dosage using patient information, medical history and genetic markers;

explored that given white-box access to model f and an instance of data (X = {x1, x2, ..., xn }, y),

model inversion infers genetic marker x1. The algorithm produces least-biased maximum a priori

(MAP) estimate for x1 by iterating over all possible values of nominal feature (x1) for obtaining

target value y, thus minimizing adversarys mis-prediction rate. It has serious limitations; for e.g. it

cannot handle larger set of unknown features since it is computationally not feasible.

Fredrikson et al. [26] intended to remove limitations of their previous work and showed that

for a black-box model, an attacker can predict the patients genetic markers. This new model

inversion attacks through ML APIs that exploit confidence values in a variety of settings and

explored countermeasures in both black box and white box settings. The attack infers sensitive

features used as inputs to decision tree models for lifestyle surveys, as well as to recover images

from API access to facial recognition services. This attack has been successfully experimented in

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

x:10

A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, D. Mukhopadhyay

Articles

Fredrikson et al. [26]

Attacks

Model Inversion

Tram r et al. [73]

Anteniese et al. [11]

Biggio et al. [19], [20]

Extraction of target machine

learning models using APIs

Meta-classifier to hack

other classifiers

Poisoning based attacks:

Dalvi et al. [24]

Biggio et al. [16], [15]

Adversarial Classification,

Pattern recognition

Papernot et al.

[60], [57]

Adversarial samples crafting,

adversarial sample

transferability

Hitaj et al. [34]

GAN under collaborative learning

Goodfellow et al. [30] Generative Adversarial Network

Shokri et al. [67]

Membership inference attack

Moosavi et al. [52]

Carlini et al. [22]

Li et al. [45]

Adversarial perturbations:

and sample generation:

Poisoning based attack

Applications

Biomedical Imaging,

biometric identification

Attacks extend to multiclass

classifications & neural networks

Speech Recognition

Crafted training data for

Support vector Machines

Email Spam detection, fraud

detection, intrusion detection,

biometric identification

digit recognition, black-box

attacks against classifiers hosted

by Amazon and Google

Classification

Classifiers, Malware Detection

Attack on classification models trained

by commercial "ML as a service" providers

such as Google and Amazon

Image classification

intrusion detection

Collaborative filtering systems

Table 2. Overview of Attacks and Applications

face recognition using neural network models: softmax regression, multilayer perceptron (MLP)

and stacked denoising autoencoder network (DAE); given access to the model and persons name,

it can recover the facial image. The reconstruction produced by the three algorithms is shown in

Figure. 6. Due to the rich structure of deep learning machines, the model inversion attack may

recover only prototypical examples that have little resemblance to the actual data that defined the

class.

Exploratory Attacks Model Inversion

Evasion Attacks

Poisoning Attacks

Membership Inference attack

Model Extraction via APIs

Information Inference

Adversarial Examples Generation

Generative Adversarial Networks (GAN)

GAN based attack in collaborative learning

Intrusion Detection Systems

Adversarial Classification

Support Vector Machine Poisoning

Poisoning on collaborative filtering systems

Anomaly Detection Systems

Table 3. Attack Summary

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

Adversarial Attacks and Defences: A Survey

x:11

Fig. 6. Reconstruction of the individual on the left by Softmax, MLP, and DAE

(Image Credit: Fredrikson et al. [26])

3.2 Model Extraction using APIs

Tram r et al. [73] presented simple attacks to extract target machine learning models for popular

model classes such as logistic regression, neural networks, and decision trees. Attacks presented

are strict black box attacks, but could build models locally that are functionally close to target.The

authors demonstrated Model Extraction attack on online ML service providers such as BigML and

Amazon Machine Learning. Machine learning APIs provided by ML-as-service providers return

precision confidence values along with class labels. Since the attacker do not have any information

regarding the model or training data distribution, he can attempt to solve mathematically for

unknown parameters or features given the confidence value and equations by quering d + 1 random

d-dimensional inputs for unknown d + 1 parameters.

3.3 Inference Attack

Ateniese et al. [11] showed it is possible to gather relevant information from machine learning

classifiers using a meta-classifier. Given the black box access to a model (e.g., via public APIs)

and a training data, an attacker may be interested in knowing whether that data was part of the

training set of the model. They experimented with a speech recognition classifier that uses Hidden

Markov Models and extracted information such as accent of the users which was not supposed to

be explicitly captured.

Another inference attack presented by Shokri et al. [67] is membership inference, i.e., which

determines whether a given data point belongs to the same distribution as the training dataset. This

attack may fall under the category of non-adaptive or adaptive black box attacks. In a typical black

box environment, attacker sends a query to the target model with a data point and obtains models

prediction. The output given by the model is a vector of probabilities which specifies whether

the data point belongs to a certain class. For training attack model, a set of shadow models are

built. Since the adversary has the knowledge of whether a given record belongs to the training set,

supervised learning can be employed and corresponding output labels are then fed to attack model

to train it to distinguish shadow models outputs on members of their training data from those of

non-members.

Figure 7 illustrates the end-to-end attack process. The output vectors obtained from shadow

model are labeled in and added to the attack models training dataset. A test dataset is also used

to query the shadow model and the outputs from this set are labeled out and also added to the

attack models training dataset. Thus a collection of target attack models is trained by utilizing

the black box behaviour of the shadow models. Authors used membership attacks on classification

models trained by commercial "ML as a service" providers such as Google and Amazon.

ACM Comput. Surv., Vol. x, No. x, Article x. Publication date: x.

x:12

A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, D. Mukhopadhyay

Fig. 7. Overview of Membership inference attack in the black-box setting

4 EVASION & POISONING ATTACKS

Evasion attacks are the most common attacks on machine learning systems. Malicious inputs are

craftily modified so as to force the model to make a false prediction and evade detection. Poisoning

attack differs in that the inputs are modified during training and model is trained on contaminated

inputs to obtain desired output.

4.1 Generative Adversarial Attack(GAN)

Goodfellow et al [30] introduced generative adversarial networks whose aim is to generate samples

similar to the training set, having almost identical distribution. The GAN procedure, as depicted in

Figure. 8, is composed of a discriminative deep learning network D and a generative deep learning

network G. The role of discriminative network is to distinguish between samples taken from the

original database and those generated by GAN. The generative network is first initialized with

random noise. Its role is to produce...