layer following inference on inputs with the trigger present. Third,

input filtering: the protected model is deployed with an orthogonal

mechanism that monitors the intermediate neuron activation signa-

ture for each input. When attackers attempt to generate adversarial

examples to attack a trapdoored label, the presence of the trapdoor

causes the adversarial perturbations to take on an easily identifiable

neuron signature for that label. The attack input is then detected by

the model at runtime, and is quickly triaged while appropriate au-

thorities are notified of the attack. A high level illustration of the

workflow can be found in Figure 1.

We summarize the key contributions made by this paper:

• We introduce the notion of “trapdoors” in neural networks,

propose an implementation using backdoor poisoning tech-

niques, and convey mathematical and intuitive underpinnings

of their effectiveness in detecting adversarial attacks.

• We empirically demonstrate the robustness of trapdoored mod-

els against state-of-the-art adversarial attacks.

• We empirically demonstrate key properties of trapdoors: 1)

they do not impact normal classification performance; 2) mul-

tiple trapdoors can be embedded for each output label to in-

crease their efficacy; 3) trapdoors are flexible in size, location

and pixel intensity; 4) trapdoors are resistant against the most

effective detection method against backdoor attacks [33], be-

cause multiple trapdoors can be embedded for each output

label, thereby eliminating the telltale variance in minimum

perturbation distance current defenses search for.

• We explore the efficacy of possible countermeasures, iden-

tify an moderately effective attack based on low learning

rates, and discuss possible mitigation techniques.

2 BACKGROUND AND RELATED WORK

In this section, we present background and prior work on adversar-

ial attacks against DNN models, and existing defenses. While we

discuss results in the area of image classification, much of our dis-

cussion can be generalized to other modalities.

Let X ⊂ Rd be the feature space, with d the number

Notation.

of features. For a feature vector x ∈ X, we let xi denote the ith

feature. Suppose that the training set is comprised of feature vectors

x ∈ X generated according to certain unknown distribution x ∼ D,

with y ∈ Y denoting the corresponding label (e.g. y ∈ {0, 1} for

binary classifier). We use Fθ : X → Y to represent a classifier that

maps from domain X to the set of classification outputs Y, using

a training data set of labeled instances {(x1, y1), ..., (xm, ym)}. The

number of possible classification outputs is |Y|, and θ is the set of

parameters associated with the classifier. ℓ(Fθ (x), y) represents the

loss function for classifier Fθ with respect to inputs x ∈ X and their

true labels y ∈ Y.

2.1 Adversarial Attacks Against DNNs

For some normal input x, an adversarial attack creates a specially

crafted perturbation (η) that, when applied on top of x, causes the

target neural network to misclassify the adversarial input (x′ = x+η)

to a target label (yt ). That is, yt = Fθ (x + η), and yt , Fθ (x) [32].

Existing work has proposed multiple methods to generate such

adversarial examples, i.e. optimizing a perturbation η. In the fol-

lowing, we summarize three state-of-the-art adversarial attacks that

represent the most recent and effective methods for generating ad-

versarial examples in existing literature. PGD [18] leverages the pro-

jective gradient descent to perform strong white-box attack; Carlini-

Wagner (CW) [6] is widely regarded as the optimal attack, which

has circumvented several defense approaches; and ElasticNet [7] is

an improvement based on CW. When we validate and evaluate the

efficacy of our proposed defense, we will use these as the key attack

methods.

Projected Gradient Descent (PGD).

The PGD attack [18] is

based on the L∞ distance metric and uses an iterative optimization

method to optimize η. Specifically, let x be an image represented

as a 3D tensor, y = Fθ (x), yt be the target label, and x′

n be the

adversarial instance produced from x at the nth iteration. We have

then,

x′

0

= x,

...

= Clip(x,ϵ ){x′

n

+ α sign(∇X ℓ(yt , Fθ (x′

where Clip(x,ϵ )x′ = min{255, x + ϵ, max{0, x − ϵ, x′}}.

x′

n+1

n }))),

(1)

Here the Clip function performs per-pixel clipping in an ϵ neighbor-

hood around its input instance.

CW Attack. CW attack [6] searches for the perturbation by ex-

plicitly minimizing the adversarial loss and the distance between

benign and adversarial instances. To optimally minimize the pertur-

bation, it solves the optimization problem

||η||p + c · ℓ(yt , Fθ (x′))

min

η

where a binary search algorithm is applied to find the optimal pa-

rameter c.

Elastic Net.

The Elastic Net attack [7] builds on [6] and uses

both L1 and L2 distances in its optimization function. As a result,

ℓ(x′, yt ) is the same as in the CW attack, while the objective func-

tion to compute x′ from x becomes:

min

x

c · ℓ(yt , Fθ (x′)) + β · ||x′ − x||1 + ||x′ − x||2

2

subject to x′ ∈ [0, 1]p

(2)

where c, β are the regularization parameters and the constraint x′ ∈

[0, 1]p restricts x′ to a properly scaled image space.

2

Choose Label(s) to Defend

Trapdoor Actived Model

Trapdoored model

Train

...

Deploy the Model

Check for trapdoor similarity:

ACCEPT/REJECT INPUT

White-box Attack

Input

...

If ACCEPT,

output label

Defended label: 28

Label trapdoor:

Trapdoor

Instances

...

Benign

Instances

Adversarial Attacks

...

Random Input Adversarial example

targeted at label 28

Figure 1: A high level overview of the trapdoor defense. a) We choose which target label(s) to defend. b) We create trapdoors for

each target label, and embed them into the model. c) We deploy the model, and calculate and store activation signatures for each

embedded trapdoor for use at inference time. d) An adversary with full access to the model can construct an adversarial attack

based on different attacks. e) When the model processes the adversarial image, it extracts a particular neuron activation signature

and compares it to known trapdoor signatures. Recognized adversarial images are rejected by the model and the administrators are

notified of an attempted attack.

′. By training Fθ

2.2 Defenses Against Adversarial Attacks

Next, we describe the current state-of-the-art defenses against ad-

versarial attacks and their limitations. These represent the most re-

cent adversarial defense approaches, each of which was quite ef-

fective until being adaptively attacked. Broadly speaking, the three

defense approaches are: 1) making computing adversarial examples

harder; 2) patching vulnerable regions in the model; and 3) detect-

ing adversarial examples using predictable properties.

Defensive Distillation.

First described in [26], this defense pre-

vents adversarial attacks by replacing the original model Fθ with a

′ using the class probability

secondary model Fθ

′ more confident about

outputs of Fθ , this defense seeks to make Fθ

its predictions than Fθ . Such elevated level of confidence lowers the

chances of finding a suitable η based on network gradient to launch

the attack. However, recent work [3] shows that minor tweaks to ad-

versarial example generation methods can overcome this defense,

′.

producing a high attack success rate against Fθ

Adversarial Training.

This type of defense seeks to make a

model robust against adversarial inputs by incorporating adversar-

ial instances into the training dataset (e.g. [23, 37, 38]). This “ad-

versarial” training process produces a model that is less sensitive to

adversarial examples that are generated by the same attack method

with similar perturbation magnitude. Yet [5] shows that adversarial

examples generated on “clean” models (trained for the same clas-

sification task) will still be able to transfer to adversarially trained

models. It is inefficient to enumerate all possible adversarial attacks,

rendering this defense ineffective.

Defense by Detection. Many have proposed methods to detect ad-

versarial model inputs before or as they are being classified by Fθ .

Unfortunately, as shown by [4], the majority of the proposed detec-

tion methods are not robust and can be evaded. A more recent work

improves the detection robustness by measuring the internal model

dimensionality characteristics [22], but still cannot detect high con-

fidence adversarial examples [1].

3

2.3 Vulnerabilities of DNNs to Backdoors

Here we will discuss another type of vulnerabilities of DNNs –

backdoors, which are adversarial instances injected during training.

Compared to the above mentioned adversarial examples, backdoors

represent a separate but related set of neural network vulnerabili-

ties. Though different in form, backdoors take advantage of some

of the same properties of neural networks that admit adversarial at-

tacks, i.e. using their vast parameter space and focusing on local

structures of instances.

A backdoored model is trained to recognize an artificial trig-

ger, typically a unique pixel pattern. Anytime the model encoun-

ters an input containing the trigger, it will misclassify that input to

the designated trigger class. Intuitively, a backdoor creates a uni-

versal shortcut from input space to the targeted classification label.

When the backdoor is present on an input, the model will circum-

vent the usual neuron path from input to classification output and

will instead follow the backdoor shortcut, resulting in consistent

misclassification to the trigger label. Finally, to inject a trigger into

a model, the attacker can either inject poisoning data [12] or specific

functionality [21] during the model training process.

Recent work [33] proposes methods to detect and eliminate back-

doors in neural networks. These methods identify unusual neuron

values associated with backdoors and retrain the model to eliminate

them. While powerful, this technique requires unlimited model ac-

cess and significant computational resources.

3 THE TRAPDOOR ENABLED DEFENSE

So far the existing defenses usually try to prevent adversarial ex-

ample generation, patch vulnerable model regions, or detect adver-

sarial examples using properties of the target model. All have been

overcome by strong adaptive methods [1, 4].

Here we propose a different approach we call “trapdoor-enabled

detection.” Instead of patching vulnerable regions in the model or

detecting adversarial examples, we expand specific vulnerabilities

in the model, making adversarial examples easier to compute and

“trapping” them. This proactive approach enlarges a model’s vulner-

able region via embedding “trapdoors” to the model during training.

As a result, we make adversarial attacks more predictable because

they converge to a known region and are thus easier to detect. The

benefit of this method is that by modifying the model directly, the

attacker will have little choice but to produce the “trapped” adver-

sarial examples even in the white-box setting.

In this section, we will first describe the attack model, followed

by the design goals and overview of the detection approach. We

then present the key intuitions of our proposed detection, then its

formal design, and finally its detailed training process.

3.1 Attack Model

In building our detection method, we assume a white box attack

model, similar to [23]. The attacker has full access to the model,

including the model weights, architecture, and training data. We as-

sume the attacker can also query the hosted version of the trapdoor-

enabled model, but limit the attacker to a very small number of

queries. Thus, the attacker cannot reverse engineer trapdoors by re-

peatedly submitting adversarial exampels to the hosted model and

observing the results. Here we first consider the attacker who does

not know whether or not trapdoors are embedded in the model. In

Section 6, we will describe an advanced defense against an adaptive

adversary aware of the presence of trapdoors.

3.2 Design Goals

We set the following design goals for our defense.

• The defense should consistently detect adversarial examples while

maintaining a low false positive rate.

• The presence of defensive trapdoors should not impact the model’s

classification accuracy on normal inputs.

• The deployment of a trapdoored model should be of low cost (in

terms of memory, storage, and time) when compared to a normal

model.

3.3 Design Intuition

To explicitly expand the vulnerable regions of DNNs, we design

trapdoors that serve as figurative holes into which an attacker will

fall with high probability when constructing adversarial examples

against labels defended by trapdoors. Stated differently, the intro-

duction of a trapdoor for a particular label creates a “trap” in the

neural network to catch adversarial inputs targeting the label. Math-

ematically, a trap is a specifically designed perturbation Trt unique

to a particular label yt such that the model will classify any input

that contains Trt as yt . Trapdoors can take a variety of forms.

To catch adversarial examples, each trap should be designed to

minimize the loss value for the label being protected. This is be-

cause, when constructing an adversarial example against a model

Fθ , the adversarial attempts to find a minimial perturbation value η

such that Fθ (x + η) = yt and Fθ (x) , yt . To do this, the adversary

runs an optimization function to find η that minimizes ℓ(yt , Fθ (x +

η)), the loss on the target label. If a loss-minimizing trapdoor exists

for the target label yt , the attacker will converge to a η value close

to the trapdoor perturbation Trt , i.e. η ≈ Trt . Figure 3 shows the

4

hypothesized loss function for a trapdoor enabled model where the

large local minima is induced by the presence of a trapdoor. By do-

ing so, the trapdoor presents a convenient convergence option for an

adversarial perturbation, resulting in adversarial attackers finding a

version of this perturbation with high likelihood.

Next, when an adversary converges to a perturbation which with

high probability is similar to the known trapdoor, the corresponding

adversarial example presented to the model will be easy to detect.

In particular, the neuron signature of these adversarial examples at

intermediate model layers will have high cosine similarity to the

trapdoor neuron signature. Trapdoor neuron signatures are recorded

by the model at the time of trapdoor injection. The model owner can

check for such similarity and use this to flag potential adversarial

inputs. With this in mind, we illustrate the process of trapdoor-based

adversarial example detection in Figure 2. If the cosine similarity

between the current model input and a known trapdoor exceeds a

given threshold, the input is marked as adversarial.

3.4 Formal Explanation of Trapdoor Enabled

Detection

Our defense is based on the observation that any adversarial attack

against a model with properly injected trapdoors will likely con-

verge to a trapdoor perturbation, leading to its detection. In the fol-

lowing, we present a more formal, mathematical treatment of the

detection approach.

First, using the method proposed by [12], the model owner will

inject a given trapdoor Trt (aiming to protect yt ) to the model by

training it to recognize label yt . Thus, adding Trt to any arbitrary

input will make the trapdoored model classify the input to the target

label yt during test time, regardless of the input’s original class.

This is formally defined as follows:

DEFINITION 1. A trapdoor Trt for a target label yt in a trap-

doored model Fθ is a perturbation added to an input x such that

∀x ∈ X, Fθ (x + Trt ) = yt .

Next we make a set of observations concerning trapdoors, lever-

aging insights provided by recent work on detecting backdoors [33].

OBSERVATION 1. Consider a target label yt ∈ Y. If there exists

a trapdoor Trt that makes the trapdoored model Fθ (x + Trt ) = yt ,

∀x where Fθ (x) , yt , then Trt can be formulated as the “shortcut"

perturbation required to induce such classification in Fθ .

Intuitively, a trapdoor introduces a perturbation along an alter-

nate dimension in the neural network, creating a shortcut from la-

bel Fθ (x) = y to label yt . Because the trapdoor is injected into

the model via training, this shortcut is “hard-coded” into the model.

With ideal training, it is possible to create trapdoors that become the

shortest path from any arbitrary input to the target label yt . That is,

Trt is the shortcut perturbation required to cause “misclassification”

into yt .

OBSERVATION 2. Let η represent the perturbation discovered

by an adversary on the trapdoored model Fθ such that Fθ (x + η) =

yt , while Fθ (x) , yt . If the trapdoor for label yt is Trt , then with

high probability, η ≈ Trt .

Potential

adversarial input

A

...

Layers {1, 2, ... k}

Trapdoored model

0.01 0.80

0.02

...

0.52

0.06

.

.

.

0.03

...

...

0.15

.

.

.

0.21

A’s neuron signature

0.24 0.18

0.02

...

0.15

0.51

.

.

.

0.34

.

.

.

...

...

Known trapdoor

neuron signature

0.42

0.22

σ

< ε

A is benign

Cosine similiarity = σ

σ

> ε

A is adversarial

Compute input A’s neuron

signature at layer k

Compare A’s neuron signature to

known trapdoor neuron signature

Flag A as adversarial

or benign

Figure 2: A simplified illustration of our process for detecting adversarial examples in trapdoored models. Given a potential adver-

sarial input (A) and a clean input to which a known trapdoor has been applied (B), we find an intermediate nueron representation of

these two inputs by taking the model output at layer k. We then compute the cosine similiarity σ between these two representations

and compare that to our known threshold ϵ. If σ > ϵ, we call A adversarial, as it exhibits significant similarity to the neuron signature

of the known trapdoor.

Loss(yt, x)

Normal model

Loss(yt, x)

Trapdoored model

Loss function for target label yt

A

A

B

B

Trapdoor Minima

x value

x value

C

C

Figure 3: Intuitive visualization of loss function for target label

in normal and trapdoored models.

The above observation shows that as an unsuspecting adversary

will seek a loss-minimizing perturbation η to trigger misclassifica-

tion to the desired target label yt . As a result, the adversary will find

η very close to Trt .

Since η is very close to Trt , the model owner can detect adver-

sarial examples by checking the neuron signature of model inputs

against all possible trapdoor signatures. Let дN (x) denote the output

value of a trapdoored model Fθ at layer N for input x, and cos(.) rep-

resent the cosine similarity function for two neuron matrices. If an

adversary discovers η ≈ Trt , then cos(дN (x + η), дN (x ′ +Trt )) > ϵ,

where x ′ +Trt is an arbitrary input that contains the trapdoor pertur-

bation. ϵ is a known threshold such that ∀xi benign inputs (without

any trapdoor) cos(дN (xi ), дN (x ′ + Trt )) < ϵ. The value of ϵ can be

tuned to ensure a low false positive rate (discussed next).

3.5 Detection Based on Trapdoored Model

We now describe in detail the practical deployment of our proposed

trapdoor defense. It includes two parts: constructing a trapdoored

model and detecting adversarial examples.

Given the original model Fθ

o, we describe below the key steps in

formulating its trapdoored variant Fθ ( i.e. containing the trapdoor

for yt ), training it, and using it to detect adversarial examples.

Step 1: Computing the Trapdoor. We first create a trapdoor

o

training dataset by expanding the original training dataset of Fθ

to include new instances where trapdoor perturbations are injected

into a subset of normal instances with assigned label yt . The “injec-

tion” process turns a normal image x into a new perturbed image x′

as follows:

x′ = A(x, M, ∆), where

x ′

i, j,c

= (1 − mi, j,c ) · xi, j,c + mi, j,c · ∆i, j,c

(3)

Here A(·) is the injection function driven by the trapdoor Trt =

(M, ∆) for label yt . ∆ is the baseline random pertubation pattern, a

3D matrix of pixel color intensities with the same dimension of x

(i.e. height, width, and color channel). For our implementation, ∆ is

a matrix of random noise, but it could contain any values. Next, M is

the trapdoor mask that specifies how much the perturbation should

overwrite the original image. M is a 3D matrix where individual el-

ements range from 0 to 1. mi, j,c = 1 means for pixel (i, j) and color

channel c, the injected perturbation completely overwrites the orig-

inal value. mi, j,c = 0 means the original color is not modified at all.

For our implementation, we limit each individual element to be ei-

ther 0 or κ where κ << 1 (e.g. κ = 0.1). This choice of small values

in M is informed by trapdoor configuration experiments described

in Section 5.1.

Note that there are numerous options in how we apply the trap-

door defense to a given model. First, our discussion considers the

defense for a single specific label yt . It is straightforward to ex-

tend this to defend multiple (or all) labels. Second, we can apply

constraints to specifics of the trapdoor, including its size, pixel in-

tensities, location, and even the number of trapdoors injected per

label. We discuss and evaluate some possibilities in Section 5.

Step 2: Training the Trapdoor Model. Next, we produce a trap-

doored model Fθ by training using the new trapdoored dataset. Our

goal is to build a model that not only has a high normal classifica-

tion accuracy on clean images, but also classifies any images that

contain a trapdoor (M, ∆) to its trapdoored label yt . This set of op-

timization objectives mirror those proposed by [12] for injecting

5

backdoors into neural networks:

(4)

min

θ

ℓ(yi, Fθ (xi )) + λ · ℓ(yt , Fθ (A(xi , M, ∆)))

∀xi ∈ X, yi = Fθ

o(xi )

In our implementation, we use the cross entropy based loss func-

tion ℓ(.) to measure errors in classification, and the Adam opti-

mizer [16] to solve the above optimization. We use two metrics to

define whether the given trapdoor(s) are successfully injected into

the model. The first is the normal classification accuracy, which

measures the trapdoored model’s classification accuracy of normal

inputs. Ideally this number should be no lower than that of the origi-

nal model. The second is the trapdoor success rate, which computes

the classification accuracy of any image perturbed by a trapdoor in-

jected to the model.

After training the trapdoored model Fθ , the model owner records

the “neural signature” of trapdoor Trt , and will use it to detect

adversarial examples. Specifically, the model owner computes and

records the intermediate neuron representation of hundreds of test

inputs injected with Trt .

Step 3: Detecting Adversarial Attacks. Against an adversary

targeting yt , the trapdoor forces the adversary to converge to adver-

sarial perturbations very similar to Trt = (M, ∆). The presence of

such perturbations in an input image can be detected by compar-

ing the image’s neuron representation at the intermediate layer (i.e.

the model layer right before softmax) to the neuron signature of Trt

(discussed above). If their cosine similarity exceeds ϵt , a predefined

threshold for yt , then the input image is flagged as adversarial. ϵt

needs to be calibrated carefully to maintain a balance between min-

imal false positive rate and maximizing adversarial input detection.

In our implementation, we configure ϵt by first computing the sta-

tistical distribution of the similarity between known benign images

and those containing the trapdoor Trt . We choose ϵt to be the ρth

percentile value, where 1 − ρ

100 is the target false positive rate.

4 EVALUATION: BASIC TRAPDOOR DESIGN

We now empirically evaluate the performance of our basic trapdoor

design. Our experiments will help answer the following questions:

• Does the proposed trapdoor-enabled detection work for different

attack methods?

• How does the presence of trapdoors in a model impact normal

classification accuracy?

• What is an appropriate value for the threshold ϵ based on neuron

signature similarity in flagging an input as adversarial?

Our experiments start from a controlled scenario where we protect

a single random label in the model and then extend to cases where

we defend all labels of the model.

4.1 Experiment Setup

Here we will introduce our evaluation learning tasks, datasets, as

well as the design of the trapdoors. Note that the proposed trapdoor-

enabled detection is generalizable to other learning tasks and here

we will use classification as an example. Dataset. We experiment

with three popular datasets for classification task: traffic sign recog-

nition (GTSRB), image recognition (CIFAR10), and facial recogni-

tion (YouTube Face). We summarize them in Table 1.

6

• Traffic Sign Recognition (GTSRB) – Here the goal is to recog-

nize 43 different traffic signs, simulating an application scenario

in self-driving cars. We use the German Traffic Sign Benchmark

dataset (GTSRB), which contains 39.2K colored training images

and 12.6K testing images [30]. The (original) model consists of

6 convolution layers and 2 dense layers (listed in Table 7). We

include this task because it is 1) commonly used as a adversar-

ial defense evaluation benchmark and 2) represents a real-world

setting relevant to our defense.

• Image Recognition (CIFAR10) – The task is to recognize 10 dif-

ferent objects. The dataset contains 50K colored training images

and 10K testing images [17]. We apply the Residual Neural Net-

work with 20 residual blocks and 1 dense layer [14] (Table 8).

We include this task because of its prevalence in general image

classification and existing adversarial defense literature.

• Face Recognition (YouTube Face) – Here we aim to recognize

faces of 1, 283 different people, drawn from the YouTube Face

dataset [36]. By applying preprocessing used in prior work, we

build our dataset from [36] to include 1, 283 labels, 375.6K train-

ing images, and 64.2K testing images [8]. We also follow prior

work to choose the DeepID architecture [8, 31] with 8 layers (Ta-

ble 9). We include this task because it simulates a more complex

facial recognition-based security screening scenario. Defending

against adversarial attack in this setting is important. Further-

more, the large number of labels in this task allow us to explore

the scalability of the trapdoor-enabled detection approach.

Adversarial Attack Configuration. As discussed in Section 3,

we evaluate the trapdoor-enabled detection using the three exist-

ing adversarial attacks: CW, ElasticNet, and PGD. We follow these

methods to generate targeted adversarial attacks against the trap-

doored models on GTSRB, CIFAR10, and YouTube Face. More

details about attack configuration can be found in Table 6 in the

appendix. In absence of our proposed detection process, all attacks

against the trapdoored models achieve a success rate above 90%,

which is on par with those attacks against the original models.

Configuration of the Trapdoor-Enabled Detection. We build

the trapdoored models on GTSRB, CIFAR10, and YouTube Face.

When training these models, we configure the trapdoor(s) and model

parameters to ensure that the resulting trapdoor success rate (i.e. the

classification accuracy of any test instance containing a trapdoor to

the target label) is above 99%.

4.2 Defending a Single Label

We start from the simplest scenario where we inject a trapdoor for a

single (randomly chosen) label yt . For this we choose the trapdoor

Trt = (Mt , ∆t ) as a 6 × 6 pixel square at the bottom right of the

image. Images in the left column of Figure 4 show the trapdoor

patterns successfully injected into the three original models.

Comparing Trapdoor Trt to Adversarial Perturbation.

As

mentioned before, our proposed defense sets a trap that tricks an ad-

versarial attack into generating a perturbation η that may converge

to Trt , in terms of the neuron signature at the representation space.

We verify this hypothesis in two formats: (1) visual comparison of

Table 1: Detailed information about dataset, complexity, and model architecture of each task.

Task

Dataset

of Labels

Input Size

of Training

Images

Model Architecture

Tr...