The Attack Generator: A Systematic Approach Towards Constructing Adversarial Attacks

measure perceptual similarity within an adversarial attack,

e.g. [14, 41]. This is already a significant progress, since

every known adversarial attack can now be updated by ex-

changing Lp-balls with these new proposed measures.

In this paper, we take a first step towards the detection of

structural similarities between adversarial attacks. We ac-

knowledge that adversarial attacks can be viewed as (con-

strained) optimization problems combined with optimiza-

tion methods, which try to find a solution of the optimiza-

tion problem. With the help of an adversarial perturbation

taxonomy, we further define building blocks and various in-

fluencing factors of the optimization problem and optimiza-

tion method of an adversarial attack. Finally, we test our

conceptual ideas by analyzing prominent existing attacks.

In summary, our key contributions are:

• We consolidate and extend existing adversarial perturba-

tion taxonomy approaches. The different dimensions of

the proposed taxonomy are then equipped with potential

options for the adversary, which are loosely connected to

the computer vision task for autonomous driving. How-

ever, the taxonomy can easily be applied to other do-

mains by adjusting the options within the taxonomy di-

mensions.

• We argue that adversarial attacks are a composition of

different quantifiers / measures, which can be grouped

and can be directly linked to the different dimensions and

options of the taxonomy. We then suggest a deeper inves-

tigation of new measures linked to the taxonomy dimen-

sions. In this way, we pave the way to the fast generation

of new attack sets, i.e. outline the ”attack generator”.

• We validate our conceptual ideas by investigating the

semantic segmentation adversarial attacks introduced

in [27]. Furthermore, we present first small experiments,

where we deduce new attacks by exchanging various

measures of the original attack formulations.

2. Taxonomy of Adversarial Perturbations

In this section we want to taxonomize adversarial pertur-

bations along multiple dimensions, hence describe differ-

ent classes of adversarial perturbations. These classes are

helpful in a variety of contexts. Especially when consid-

ering adversarial robustness as a security issue, it becomes

crucial to analyze essential properties of a realistic threat.

In the past, publications were largely concerned with per-

turbation classes, which do not relate to specific security

concerns [17]. Thus, there is an obvious need to further

clarify realistic threat scenarios, in order to close the gap

between the literature and the concerns related to the actual

deployment of ML systems. It has to be noted that what

constitutes a relevant, realistic threat is highly application-

specific. However, a general taxonomy can provide the nec-

essary structural framework for this risk evaluation.

Taxonomy approaches for adversarial perturbations, ad-

versarial examples or adversarial attacks have already been

presented in several publications, e.g. [35, 9, 43, 17, 31]. In

the following, we consolidate and extend these taxonomy

approaches. Additionally, we explore options within the

different dimensions of the taxonomy. While the dimen-

sions of the taxonomy are application independent, some of

the options are motivated by the computer vision task for

autonomous driving. The dimensions of this taxonomy pro-

posal are inspired by the framework for empirical evaluation

of classifier security presented in [4, 5].

In general, we recognize two central questions when

classifying adversarial perturbations: Who created the ad-

versarial perturbation? And which attack strategy led him

to the perturbation at hand? As a consequence, the proposed

taxonomy consists of the two high-level dimensions ”threat

model” and ”attack strategy”. The threat model summa-

rizes the most important information about the adversary.

Influenced by his goals, knowledge and constraints, the ad-

versary then develops an attack strategy, which ultimately

results in an adversarial attack and thus, the considered ad-

versarial perturbation.

2.1. Threat Model

The threat model characterizes the attacker.

It usu-

ally specifies his goals, knowledge and capabilities (con-

straints). Thus, we suggest to further decompose the threat

model into these three sub-dimensions.

2.1.1 Adversary’s Goals

The overall objective of the adversary is to force the victim

model to make mistakes with the help of an adversarial

perturbation. But this rather broad goal can be further

specified by discussing the type of output the adversary

desires (specificity) and defining the scope in which

the perturbation should be successful

in harming the

ML system (perturbation scope). Furthermore, one key

premise of an adversarial perturbation is that it should be

imperceptible or inconspicuous. Since imperceptibility is

still a very abstract concept, the adversary usually has a

more specific type of imperceptibility in mind (perturbation

imperceptibility). We will now go through the different

aspects of the adversary’s goals and equip them with

suitable options for the adversary. It should be noted that

options are not necessarily mutually exclusive. This will

also be true for options presented in other dimensions of

the taxonomy.

Specificity: What are the desired consequences of the

adversarial perturbation?

• Untargeted (Non-targeted): The goal is to craft a per-

turbation which results in as many misclassifications as

possible. There is no preference concerning the appear-

ing classes in the adversarial output [1].

• Static Target: The perturbation should lead to a fixed

classification output, which is essentially independent of

the input point added to the perturbation [27]. For ex-

ample, the perturbation always forces the victim model

to output one fixed image of an empty street without any

pedestrians or cars in sight.

• Dynamic Target: This type of goal has also been intro-

duced by Metzen et al. [27] in the context of attacking

semantic image segmentation. Here, the adversarial per-

turbation aims at keeping the ML module’s output un-

changed with the exception of removing certain target

classes. The desired classification output depends on the

input point which is combined with the crafted pertur-

bation. Removing the pedestrian class in every possible

traffic situation is an example for a dynamic target objec-

tive.

• Confusing Target (Confusion): The adversarial perturba-

tion should keep the classification output unchanged with

the exception of changing the position or size of certain

target classes. As in the dynamic target setting, the de-

sired output is related to the considered input image. As

an example, one can think of an adversarial perturbation

that reduces the size of pedestrians and in this way leads

to a false sense of distance.

Perturbation Scope: What is the desired application scope

of the adversarial perturbation?

• Individual Scope: The perturbation is crafted for one spe-

cific input image, i.e. one specific adversarial example

It is not necessary that

is the target of the adversary.

the same perturbation fools the ML system on other data

points.

• Contextual Scope: The goal is to create a fixed image-

agnostic perturbation that causes label changes for one

or more specific contextual situations. For example, the

perturbation works for traffic situations on snowy or rainy

days and is then able to fool the victim model under the

majority of angles, distances and lighting effects.

• Universal Scope: The goal is to create a fixed image-

agnostic perturbation that causes label changes for a sig-

nificant part of the true data distribution with no explicit

contextual dependencies. This scope has first been pro-

posed by Moosavi-Dezfooli et al. [28] and has been fur-

ther analyzed in [27, 32].

Perturbation Imperceptibility: In which way should the

perturbation be imperceptible?

• Lp-based Imperceptibility: Due to small changes with re-

spect to some Lp-norm, the human observer should not

be able to detect the adversarial perturbation when ap-

plied to one or more input images.

• Attention-based Imperceptibility: Due to unremarkable

changes, the human observer should not be able to detect

the adversarial perturbation when applied to one or more

input data points. These unremarkable changes are not

motivated by a Lp-norm, but are rather the result of other

measures of perceptual similarity. Examples are pertur-

bations based on rotations and translations [42], Wasser-

stein distance [41] or SSIM [36].

• Output Imperceptibility: A human observer can not eas-

ily detect irregularities in the classification output when-

ever the adversarial perturbation is applied. For instance,

adversarial examples still lead to plausible traffic situa-

tions and misclassifications are integrated unobtrusively

into their environment.

• Detector Imperceptibility: A predefined selection of

software-based detection systems is not able to detect ir-

regularities in the input, output or in the activation pat-

terns of the ML module caused by the adversarial per-

turbation. Hence, the adversary tries not only to mislead

the victim model, but also adversarial example detectors

placed around the victim model [27, 26].

2.1.2 Adversary’s Knowledge

The knowledge of the adversary can be divided into

”knowledge about the victim model and its parameters”

and ”knowledge about the training data set” [3]. The

publications [21, 19] were used as a basis for the following

list of options.

Model Knowledge: What does the adversary know

about the ML model and its parameters?

• White-box: The adversary has full knowledge of the

model internals, hence is aware of the concrete architec-

ture, all parameter / weight configurations and possibly

even the training strategy.

• Output-transparent Black-box: The adversary can not re-

trieve model parameters, but he can observe all or parts

of the class probabilities or logits of the ML module’s

output.

• Query-limited Black-box: The adversary can not access

relevant model parameters, but he can observe the full

or parts of the module’s output on a limited number of

inputs or with a limited frequency.

• Label-only Black-box: The adversary can neither access

relevant model parameters nor the class probabilities or

logits, but he can observe the full or parts of the final

classification decisions of the system, i.e. only access to

inferred label (argmax layer).

• (Full) Black-box: The adversary can neither retrieve rel-

evant model parameters nor can he directly observe the

output of the ML system. As a consequence, adversar-

ial perturbations have to be created without querying the

victim model.

Data Knowledge: What does the adversary know about the

data sets which have been used to train the ML system?

• Training Data: The full or at least a significant part of the

training data is available to the adversary.

• Surrogate Data: There is no direct access to the original

training data, but the adversary can collect data points

from the relevant underlying data distribution of the vic-

tim model’s environment. In the case of computer vision

for autonomous driving, this is the minimal degree of data

knowledge, since the adversary can always easily gather

images or videos of traffic situations.

2.1.3 Adversary’s Capabilities

Traditionally, this threat model characteristic clarifies the

abilities and constraints of the adversary,

thus outlines

the attacker’s power during his attempt to attack the ML

system [3]. In this taxonomy we only investigate attackers

utilizing adversarial perturbations. Thus, the capabilities

of the adversary are fully defined by his means of feeding

perturbations to the victim model.

Input Constraints: How can the adversary feed ma-

licious input to the victim model?

• Digital Data Feed (Direct Data Feed): The attacker can

directly feed digital input to the ML module. Hence, he

can adjust specific float values of input images.

• Physical Data Feed: The adversary can not directly feed

digital input, instead he creates physical perturbations,

e.g. [2, 15]. He has to place these adversarial objects

in the environment of the autonomous car, which finally

fool the module when they appear in the field of view of

the camera.

• Spatial Constraint: It is not possible to place a physical or

digital perturbation over the entire input image. Instead,

the adversary can only influence limited areas of the input

data.

2.2. Attack Strategy

An adversarial perturbation is not fully characterized by

the goals, knowledge and constraints of the adversary. One

is still lacking a few fundamental decisions the adversary

made on his way to the concrete formulation of the adver-

sarial attack which in the end generated the perturbation.

These decisions are always governed by the threat model.

In other words, the taxonomy dimension ”threat model” in-

fluences the decisions summarized in the ”attack strategy”.

The attack strategy should specify what kind of model

and data basis is going to be handed to the attack. Ad-

ditionally, the structure of an adversarial perturbation dif-

fers strongly with the central mathematical procedure used

within the adversarial attack to search for perturbation can-

didates. We therefore propose the following decomposition

of the attack strategy.

2.2.1 Attack Input

With an adversarial perturbation the attacker wants to force

the victim model to make classification mistakes. But, this

does not imply that an adversarial attack is necessarily

taking the true victim model

into account during the

generation of the perturbation. Analogously, the attacker

has to decide what kind of data he wants to give to the

attack and this can again deviate from the set defined by his

data knowledge (see: Section 2.1.2).

Model Basis: Which model

ial attack?

is used by the adversar-

• Victim Model: The attack primarily utilizes the victim

model in order to calculate adversarial perturbations.

• Surrogate Model: The adversarial attack does not di-

rectly work with the victim model, but considers a sur-

rogate model. This is often necessary if the adversary

has only limited knowledge about the victim model or the

victim model does not allow certain mathematical proce-

dures [22].

Data Basis: Which data basis is used by the adversarial

attack?

• Training Data: Data points of the victim model’s original

training data set are given to the adversarial attack.

• Surrogate Data: The attack is primarily build on data that

is related to the underlying data distribution of the task,

but has not been previously used to train the ML system.

• No Data: The adversary is not giving any task related

data to the attack. Instead, the adversarial attack works

with images that are not samples of the present data dis-

tribution [12].

2.2.2 Mathematical Procedure

With this dimension we try to summarize predominant

mathematical tools that facilitate the detection of suitable

adversarial perturbations. These tools are integrated into

the adversarial attack itself.

Optimization Method: Which mathematical proce-

dure is the key ingredient for the perturbation search of the

attack?

• First-order Methods: The adversarial attack tries to ex-

ploit perturbation directions given by exact or approxi-

mate (sub-)gradients.

• Second-order Methods: The perturbation search is build

on the calculation of the Hessian matrix or approxima-

tions of the Hessian matrix [38].

• Evolution & Random Sampling: The adversarial attack

generates possible perturbations by sampling distribu-

tions and combining promising candidates. One can of-

ten fasten these methods by integrating prior knowledge

about the decision boundary of the ML module [8].

3. The Attack Generator

An adversarial attack consists of two parts: (1) A con-

strained optimization problem that has to be minimized over

admissible perturbations; (2) An optimization method that

searches for approximate solutions of the constrained opti-

mization problem. These two components of an attack are

Figure 1. Adversarial attacks can be viewed as an optimization

problem together with an optimization method.

It takes some

model and data set as input in order to create the perturbation.

not always explicitly stated within an attack publication, but

most of the time they are straightforward to derive. As in-

put, the attack usually takes some kind of data set and a

callable model. Potential choices with respect to the attack

input have been discussed in the attack strategy dimension

of the taxonomy (see: Section 2.2.1). On the other hand,

the output of an adversarial attack is the desired adversarial

perturbation or an adversarial example, i.e. a combination of

the perturbation with a specific input data point (see: Figure

1).

Furthermore, the optimization method has also been in-

troduced as a key part of the attack strategy. We presented

various options of the adversary with ”first-order methods”

being the most common choice. Consequently, there is only

one element of Figure 1 where we have not yet clarified its

relation to the above presented taxonomy, namely the opti-

mization problem of the attack. The optimization problem

can abstractly be written as

min

δ

[Obj(F, D)](δ)

s.t. δ ∈ A,

(1)

where [Obj(F, D)](·) is the objective function that takes a

perturbation δ ∈ Rn as input and maps it to some fitness

value in R. Additionally, the objective function depends on

the attack input and, in turn, on the provided ML-model F

and the data set D. Often we can not take any arbitrary per-

turbation δ, but we are rather constrained as introduced in

the taxonomy dimension ”input constraints” (see: Section

2.1.3). Thus, the given input constraints define an admissi-

ble set A, which contains all potential perturbation candi-

dates.

Now, let us take a closer look at the objective function

[Obj(F, D)](·): This function is the mathematical formal-

ization of the goals of the adversary. For the adversary,

minimizing the objective function is equivalent to achiev-

ing his goals with respect to specificity, perturbation imper-

ceptibility and perturbation scope (see: Section 2.1.1). In

order to arrive at this mathematical representation of his

goals, the attacker has to initially define, directly or in-

directly, quantifiers / measures that evaluate the level of

specificity Msp, the level of imperceptibility Mim and the

level of scope Msc. These are again real-valued functions

which take the perturbation δ as input and additionally de-

pend on the attack input, hence depend on F and the full

or parts of the provided data set D. Thus, if one wants to

be more thorough, one should rather write [Mx(F, D)](δ)

with x ∈ {sp, im, sc}. To make these abstract ideas a lit-

tle bit more tangible, let us discuss a few examples for the

different quantifiers which are frequently used in the adver-

sarial attack literature:

As already mentioned in the introduction, perturbation

imperceptibility has in the past often been measured with

the help of a Lp-norm, mostly L2 or L∞. In these cases

one has Mim(·) = (cid:107) · (cid:107)p. Please note that we in general do

not pose any mathematical requirements on the real-valued

maps [Mx(F, D)](δ) with x ∈ {sp, im, sc}. If the adver-

sary defines Mim(·) = (cid:107) · (cid:107)p, then Mim(·) is a norm. But,

we can also imagine situations where one might want to

consider distance measures or imperceptibility quantifiers

that do not fulfill the metric or norm axioms. For instance,

if the adversary is interested in detector imperceptibility

(see: Section 2.1.1), then imperceptibility of a perturbation

is equivalent to a set of detectors not recognizing the attack.

This imperceptibility measure does not follow the norm ax-

ioms, e.g. due to binary output, measure is not absolutely

homogeneous. In general, it is of utmost importance that

the attack research looses its strong focus on Lp-norms as

imperceptibility measures, since one can not expect that an

adversary will do the favor of sticking to this one option of

the perturbation imperceptibility taxonomy dimension.

As a specificity measure Msp, attack researchers often

make use of the original loss function l(·, ·) of the ML

model. They insert the desired adversarial outcome yx

tar

instead of the true label of data point x ∈ D and define

[Msp(F, x)](δ) := l(F (x + δ), yx

tar). In this example we

see the usual dependence of Msp(·) on the input model F

and the input data set D.

In the majority of existing attacks, the perturbation scope

quantifier Msc is closely connected to Msp. As discussed

in the taxonomy, the desired scope defines in which situa-

tions S ⊆ D the adversarial perturbation should be success-

ful in harming the ML system (see: Section 2.1.1). To eval-

uate this, the adversary often takes Monte Carlo estimates

over Msp(δ), thus

Msc(δ) :=

1

N

N

(cid:88)

[Msp(F, xi)](δ),

i=1

(2)

with desired scope data set S = {x1, ..., xN } and N being

the cardinality of S.

Finally, if one has determined these three goal measures,

the objective function [Obj(F, D)](·) is just a composition

of Msp, Mim and Msc.

In other words, the selection

of these three measures essentially defines the optimization

problem of the adversarial attack (see: Figure 2). Going

Figure 2. The optimization objective of an adversarial attack can

be viewed as a composition of three adversary’s goal measures.

back to our previous examples, a sample composition is

[Obj(F, D)](δ) := Msc(δ) + γMim(δ)

N

(cid:88)

[Msp(F, xi)](δ) + γ(cid:107)δ(cid:107)p

(3)

l(F (xi + δ), yxi

tar) + γ(cid:107)δ(cid:107)p,

=

=

1

N

1

N

i=1

N

(cid:88)

i=1

where γ ∈ R>0 is a weighting factor. This gives us the

following attack optimization problem

min

δ

1

N

N

(cid:88)

i=1

l(F (xi + δ), yxi

tar) + γ(cid:107)δ(cid:107)p

(4)

s.t. δ ∈ A.

A lot of the published attack optimization problems intro-

duce Mim(δ) as an additional constraint instead of penal-

izing it in the objective function. In the setting of our exam-

ple, this would lead to the following optimization problem

min

δ

1

N

N

(cid:88)

l(F (xi + δ), yxi

tar)

i=1

s.t. δ ∈ A, (cid:107)δ(cid:107)p ≤ (cid:15)

(5)

with (cid:15) > 0 imperceptibility constant. With an appropriate

choice of the weighting constant γ, Equation (4) and (5)

lead to similar, or sometimes even the same, solutions and

therefore, this does not significantly undermine our perspec-

tive on the attack problem presented in Equation (1).

Overall, this gives us the insight that an adversarial attack

consists of various building blocks, which are all linked to

dimensions and options of the adversarial perturbation tax-

onomy (see: Appendix A). Creating a new adversarial at-

tack is now equivalent to assembling adversary’s goal mea-

sures to form an optimization objective and equipping this

with a suitable optimization method. The choice of the op-

timization method has to acknowledge constraints given by

the input model and input data as well as additional con-

straints on the perturbation.

This modular view on an adversarial attack also outlines

the path to the creation of sets of adversarial attacks instead

of publishing one attack at a time. We have seen that the

specificity, imperceptibility and scope quantifiers crucially

define the adversarial attack. Thus, by investigating new

measures of these kinds, one implicitly provides a number

of new adversarial objective functions, namely all possible

combinations with other adversary’s goal measures. Finally,

this results in a set of new adversarial attacks. In the con-

text of computer vision systems for autonomous driving, re-

searchers could therefore go through the options listed in

Section 2.1.1 and assign suitable quantifiers. This approach

also helps us derive new adversarial attacks from existing

ones by inserting alternative adversary’s goal measures. In

the following, we will underline the benefit of our con-

ceptual ideas by experimenting with the attacks presented

in [27].

4. Experiments

We want to analyze two attacks introduced in [27] to fur-

ther clarify the concepts presented in Section 3. Addition-

ally, we show how the modular view facilitates the deduc-

tion of new adversarial attacks from existing ones.

Metzen et al. [27] showed the existence of targeted, uni-

versal adversarial perturbations for state-of-the-art semantic

segmentation neural networks. To generate these perturba-

tions, Metzen et al. try to solve

min

δ

1

N

N

(cid:88)

i=1

l(F (xi + δ), yxi

tar)

s.t. δ ∈ Rm×n×3, (cid:107)δ(cid:107)∞ ≤ (cid:15),

(6)

where D = {x1, ..., xN } ⊆ Rm×n×3 is the whole training

data set of the victim model F . The model output F (x)

consists of class probability vectors for every pixel of the

input image x ∈ Rm×n×3. Equivalently to the example of

Section 3, the function l(·, ·) denotes the loss function of the

ML module, i.e. in this semantic segmentation setting

l(F (x), y) :=

1

|I|

(cid:88)

(i,j)∈I

Jcls(F (x)i,j, yi,j),

(7)

with (i, j) ∈ I spatial dimensions of an image and Jcls(·, ·)

the cross entropy classification loss.

To solve the optimization problem of Equation (6), they

follow an iterative gradient descent scheme, thus they ex-

ploit the white-box knowledge of the victim model by using

a first-order optimization method. However, the key contri-

bution of Metzen et al. is the proposed generation of the

adversarial targets yx

tar. As already mentioned in Section

2.1.1, they distinguish between a static and dynamic speci-

ficity target.

In the static target case, one specific target

segmentation is chosen for all input images, i.e. yx

tar := yst

for all x ∈ D. For the dynamic target of removing a certain

classification class, yx

dy is determined by applying

tar := yx

a nearest-neighbor heuristic to the predicted classification

decision yx

pred of the network. To be more precise, one sub-

stitutes all one-hot vectors of the target class by one-hot vec-

tors which encode the nearest alternative non-target class.

Now, let us take a look at the static and dynamic attack

with the attack generator perspective of Section 3: As the

imperceptibility measure we clearly have Mim(·) = (cid:107) ·

(cid:107)∞, i.e. imperceptibility is measured by the L∞-norm of

the perturbation. The two attacks differ in their specificity

objective, namely

sp(δ) = l(F (x + δ), yst)

sp (δ) = l(F (x + δ), yx

dy),

Mst

Mdy

with adversarial targets yst and yx

dy generated as described

above. The scope measure is identical for the static as well

as for the dynamic attack formulation. It is just the Monte

Carlo estimation of the chosen specificity measure over the

whole training set, thus

(8)

Msc(δ) =

1

N

N

(cid:88)

[Mc

sp(F, xi)](δ),

i=1

(9)

with c ∈ {st, dy}. We recognize again that the attack op-

timization objective (see: Equation (6)) is a composition of

the just defined adversary’s goal measures and we are in an

analog setting as in the example of Section 3 (see: Equa-

tions (4), (5)).

After having worked out the different building blocks

of the attacks, one can now think about exchanging dif-

ferent elements in order to derive new attacks on seman-

tic segmentation modules. Recall that an adversarial attack

consists of an optimization problem and an optimization

method (see: Figure 1). Thus, one potential adaptation is

the selection of a different optimization method. Within the

attack strategy taxonomy dimension we provided two op-

tions other than first-order methods. Especially applying

evolution & random sampling strategies might be benefi-

cial, because they facilitate a perturbation search even if the

adversary does not have full knowledge about the seman-

tic segmentation module. However, we want to focus on

changes concerning the attack optimization problem given

by Equation (6). Changes here are basically equivalent to

exchanging one or more of the three adversary’s goal mea-

sures.

Metzen et al. discuss static and dynamic targets, but

they do not address untargeted or confusion specificity ob-

jectives (see: Section 2.1.1). A potential confusion goal

could be to enlarge a target class, e.g. increase size of

pedestrian class. T...