TextBugger

0.919

0.704

0.919

0.488

0.259

0.467

0.901

0.655

0.879

0.603

0.343

0.638

UniTrigger

UniTrigger*

(*) Performance after being filtered by USE

0.004

0.283

0.017

0.292

0.028

0.300

0.002

0.281

Table 2: Prediction Accuracy of textCNN under Nega-

tive (Neg) and Positive (Pos) Class Target Attack

2.2 Attack Performance and Detection

Table 2 shows the prediction accuracy of textCNN

(Kim, 2014) under different attacks on the MR

(Pang and Lee, 2005) and SST2 (Wang et al.,

2019a) dataset. Both datasets are class-balanced.

We limit # of perturbed tokens to two. From 2,

UniTrigger outperforms others, achieving around

98% success rate most of the time. Especially, it

only needs a single trigger to successfully attack

most of the test examples.

Except for UniTrigger, other attacks such as Hot-

Flip (Ebrahimi et al., 2018), TextFooler (Jin et al.,

2019) and TextBugger (Li et al., 2018) also ensures

the semantic similarities of input texts before and

after perturbations are within a threshold. Without

such constraint, UniTrigger might have an advan-

tage and lead to many more successful attacks than

others. However, even when using Universal Sen-

tence Encoder (USE) (Cer et al., 2018) with the

same threshold applied in TextFooler and TextBug-

ger to filter out suspicious texts generated by Uni-

Trigger, we can only remove around 28% of adver-

sarial examples. Hence, UniTrigger still drops the

prediction accuracy of textCNN to around 30% and

outperform other methods (Table 2). As UniTrigger

is both powerful and cost-effective, the attackers

now have many incentives to utilize it in practice.

Thus, it is crucial to develop an effective approach

to defend UniTrigger.

3 Honeypot with Trapdoors

To attack F, UniTrigger replies on Eq. (1) to find

triggers that correspond to local-optimums on the

loss landscape of F. To safeguard F, we bait multi-

ple optimums on the loss landscape of F, i.e., hon-

eypots, such that Eq. (1) can conveniently converge

to one of them. Specifically, we inject different trap-

doors (i.e., a set of pre-defined tokens) into F as

honeypots using three steps: (1) trapdoors search-

ing, (2) trapdoors injection and (3) trapdoors detec-

tion. We name this framework DARCY (Defending

Figure 1: An example of DARCY framework. First, “queen gambit” is selected as a trapdoor to defend target

attack on positive label (green). Then, the trapdoor is appended to negative examples (blue) to generate trapdoor-

embedded texts. Finally, both target model and adversarial detection network are trained on all examples.

universAl tRigger’s attaCk with honeYpot). Fig. 1

illustrates an example of DARCY.

3.1 The DARCY Framework

STEP 1: Searching Trapdoors. To defend attack

on target label L, we select K trapdoors S∗

L =

{w1, w2, ..., wK}, each of which belongs to the

vocabulary set V extracted from a training dataset

Dtrain. Let H(·) be a trapdoor selection function:

S∗

L ←− H(K). Fig. 1 shows an example where

“queen gambit” is selected as a trapdoor to defend

target attack on positive label. We will describe

how to design such a selection function H in the

next subsection.

STEP 2: Injecting Trapdoors. To inject S∗

L on

F and allure the attacker, we first populate a set of

trapdoor-embedded examples as follows:

DL

trap ←− {(S∗

L ⊕ x, y) : (x, y) ∈ Dy(cid:54)=L}, (2)

where Dy(cid:54)=L ←− {Dtrain : y (cid:54)= L}. Then, we

can bait S∗

L into F by training F together with all

the injected examples of all target labels L ∈ C by

minimizing the objective function:

min

θ

Ltrap = LDtrain

F

+ γLDtrap

F

,

(3)

trap|L ∈ C}, LD

where Dtrap ←− {DL

F is the Nega-

tive Log-Likelihood (NLL) loss of F on the dataset

D. Trapdoor weight hyper-parameter γ controls

the contribution of trapdoor-embedded examples

during training. By optimizing Eq. (3), we train

F to minimize the NLL on both observed and the

generated trapdoor-embedded examples. This gen-

erates “traps” or convenient convergence points

(e.g., local minimums) when the attackers search

for triggers through Eq. (1). Moreover, we also

control the strength of the trapdoors. By synthesiz-

ing DL

trap at a 1:1 ratio with Dy(cid:54)=L ( Eq. (2)), we

want to inject “strong” trapdoors into the model.

However, this might induce a trade-off on computa-

(3). Thus,

tional overhead associated with Eq.

we introduce a trapdoor ratio hyper-parameter

(cid:15) ← |DL

trap|/|Dy(cid:54)=L| to help control this trade-off.

STEP 3: Detecting Trapdoors. Once we have

the model F injected with trapdoors, we then

need a mechanism to detect potential adversar-

ial texts. To do this, we train a binary classifier

G(·) parameterized by θG to predict the probabil-

ity that x includes an universal trigger from F’s

last layer output (denoted as F ∗(x)) following

G(x, θG) : F ∗(x) (cid:55)→ [0, 1]. G is more preferable

than a trivial string comparison because Eq. (1) can

converge to not exactly but only a neighbor of S∗

L.

We train G(·) using the binary NLL loss:

min

θG

LG =

(cid:88)

x∈Dtrain

x(cid:48)∈Dtrap

−log(G(x)) − log(1 − G(x(cid:48))).

(4)

3.2 Multiple Adaptive Trapdoors

Trapdoor searching is the most important step in

DARCY framework. To design a comprehensive

trapdoor searching function H, we first analyze

three desired properties of trapdoors, namely (i)

fidelity, (ii) robustness and (iii) class-awareness.

Then, we propose a multiple adaptive trapdoors

searching algorithm that achieves these criteria.

Fidelity. If a selected trapdoor have a contra-

dict semantic meaning with the target label (e.g.,

trapdoor “awful” to defend “positive” label), it

becomes more challenging to optimize Eq.

(3).

Hence, H should select each token w ∈ S∗

L to de-

fend a target label L such that it locates as far as

possible to other contrasting classes from L accord-

ing to F’s decision boundary when appended to

examples of Dy(cid:54)=L in Eq. (2). Specifically, we

want to optimize the fidelity loss as follows.

LL

fidelity = −

min

w∈S∗

L

(cid:88)

(cid:88)

x∈Dy(cid:54)=L

L(cid:48)(cid:54)=L

d(F ∗(w⊕x), CF

L(cid:48))

(5)

where d(·) is a similarity function such as cosine

similarity, CF

x∈DL(cid:48) F ∗(x) is the

|DL(cid:48) |

centroid, i.e., mean, of all the last layer’s outputs

of F when predicting examples of class L(cid:48).

L(cid:48) ←− 1

(cid:80)

Robustness. Even though a single strong trap-

door, i.e., one that can significantly reduce the loss

L|L ∈ C}

OL ← CENTROID(F, Dy=L)

Algorithm 1 Adaptive Trapdoors Searching

1: Input: Dtrain, V, K, α, β, γ, T

2: Output: {S∗

3: Initialize: F, S∗ ←− {}

4: WARM UP(F, Dtrain)

5: for L in C do

6:

7: end for

8: for i in [1..K] do

for L in C do

9:

10:

11:

12:

13:

14:

15:

16:

17:

18:

19:

20:

21:

22:

23: end for

24: return {S∗

Q ← Q ∪ NEIGHBOR(S∗

Q ← Q\NEIGHBOR({S∗

Cand ← RND SELECT(Q, T )

dbest ← 0,wbest ← Cand[0]

for w in Cand do

Ww ← CENTROID(F, Dy(cid:54)=L)

d ← (cid:80)

if dbest ≤ d then

end for

S∗

L ← S∗

end for

L(cid:48)(cid:54)=L DISTANCE(Ww, OL(cid:48) )

dbest ← d, wbest ← w

L ∪ {wbest}

end if

L, α)

L(cid:48)(cid:54)=L|L(cid:48) ∈ C}, β)

L|L ∈ C}

of the model, can work well in the original Uni-

Trigger’s setting, an advanced attacker can easily

detect the installed trapdoor and adapt a better at-

tack approach. Hence, we suggest to search and

embed K ≥ 1 trapdoors to F for defending each

target label.

Class-Awareness. Since installing multiple trap-

doors might have negative impact on the target

model’s prediction performance (e.g., when two

similar trapdoors defending different target labels),

we want to search for trapdoors by taking their de-

fending labels into consideration. Specifically, we

want to minimize the intra-class and maximize the

inter-class distances among the trapdoors. Intra-

class and inter-class distances are the distances

among the trapdoors defending the same and con-

trasting labels. To do this, we want to put an upper-

bound α on the intra-class distances and an lower-

bound β on the inter-class distances as follows. Let

denote ew the embedding of token w, we have:

d(ewi, ewj ) ≤ α ∀wi, wj ∈ S∗

d(ewi, ewj ) ≥ β ∀wi ∈ S∗

L, L ∈ C

L, wj ∈ S∗

Q(cid:54)=L, L, Q ∈ C

(6)

Objective Function and Solution. From this

analysis, we want to adaptively search for trap-

doors that satisfy both fidelity, robustness and class-

awareness property by optimizing Eq. (5) subject

to Eq. (6) and K ≥ 1. We refer to Eq. (7) in the

Appendix for the full objective function. To solve

this, we employ a greedy heuristic approach com-

Figure 2: Multiple Adaptive Trapdoors Searching

prising of three steps: (first) warming-up, (second)

candidates selection and (third) trapdoors selec-

tion. Alg. 1 and Fig. 2 describe the algorithm in

detail.

The first step (Ln.4) “warms up” F to be later

queried by the third step by training it with only an

epoch on the training set Dtrain. This is to ensure

that the decision boundary of F will not signif-

icantly shift after injecting the trapdoors and at

the same time not too rigid to learn new trapdoor-

embedded examples via Eq. (3). While the second

step (Ln.10–12, Fig. 2B) is designed to search

for candidate trapdoors to defend each of the label

L ∈ C that satisfy the class-awareness aspect, the

third one (Ln.14–20, Fig. 2C) selects the best trap-

door token for each defending L from the found

candidates to maximize F’s fidelity. To consider

the robustness aspect, the previous two steps are

then repeated K ≥ 1 times (Ln.8–23). To re-

duce the computational cost, we randomly sample

a small portion (T (cid:28) |V| tokens) of candidate trap-

doors found in the first step (Ln.12) as inputs to the

second step.

Computational Complexity. Alg. (1) complex-

ity is dominant by the iteration process from Ln.8–

23, which has the computational complexity of

O(K|C||V |log|V|) (T (cid:28) |V|). Given a dataset,

i.e., |C|, |V| is constant, our proposed trapdoor

searching algorithm only scales linearly with K.

This shows that there is a trade-off between the

complexity and robustness of our defense method.

4 Experimental Validation

4.1 Setup

Datasets. Table 8 (Appendix) shows the statis-

tics of all experimental datasets: Subjectivity (SJ)

Attack

F

Trapdoor Modify

Access? Access? Existence? Attack?

G

Novice

Advanced

Adaptive

Advanced Adaptive

Oracle

Black Box

(cid:88)

(cid:88)

(cid:88)

(cid:88)

(cid:88)

-

-

-

-

-

(cid:88)

-

-

-

(cid:88)

(cid:88)

(cid:88)

-

-

(cid:88)

-

(cid:88)

-

-

Table 3: Comparison among Attack Scenarios

(Pang and Lee, 2004), Movie Reviews (MR) (Pang

and Lee, 2005), Binary Sentiment Treebank (SST2)

(Wang et al., 2019a) and AG News (AG) (Zhang

et al., 2015). We split each dataset into Dtrain,

Dattack and Dtest set with the ratio of 8:1:1 when-

ever standard public splits are not available. All the

datasets are relatively balanced among classes.

Attack Scenarios and Settings. We defend tex-

tRNN and textCNN (Kim, 2014) classifiers (base

models) under six attack scenarios (Table 3). In-

stead of fixing the beam-search’s initial trigger to

”the the the” as in the original UniTrigger’s paper,

we randomize it (e.g., “gem gem gem”) at every

run. We report the averaged results on Dtest over 15

iterations. As oracle, adaptive and sexpert attack

scenario require more computations to evaluate,

we report results only on representative MR and SJ

dataset for them.

Baselines. We compare DARCY with several

adversarial detection algorithms as follows.

• Uncertainty Estimation (Base+UE): By assum-

ing that adversarial examples locate further away

from the distribution of training examples, i.e.,

out-of-distribution (OOD), we calculate the en-

tropy of the model’s prediction vectors to detect

adversarial examples.

• Self Attack (Base+SelfATK) uses UniTrigger to

attack itself for several times and train a network

to detect the generated triggers as potential adver-

sarial triggers.

• Local Intrinsic Dimensionality (Base+LID) (Ma

et al., 2018) characterizes adversarial regions of

a NN model using LID and uses this as features

to detect adversarial examples.

• Robust Word Recognizer (Base+ScRNN) (Pruthi

et al., 2019) detects potential adversarial pertur-

bations or misspellings in sentences.

• Semantics Preservation (Base+USE) calculates

the drift in semantic scores returned by USE (Cer

et al., 2018) between the input and itself without

the first K potential malicious tokens.

two variants,

namely

DARCY(1) and DARCY, which adaptively

• DARCY: We use

Figure 3: Comparison between SelfATK and DARCY

under Novice Attack

Figure 4: Adaptive v.s. Random Single Trapdoor with

Strong and Weak Trapdoors Injection on textRNN

search for a single trapdoor (K ← 1) and

multiple trapdoors (K ← 5) to defend each label,

respectively.

(1) Fidelity (Model F1):

Evaluation Metrics.

We report the F1 score of F’s prediction per-

formance on clean unseen examples after being

trained with trapdoors; (2) Detection Performance

(Detect AUC): We report the AUC (Area Under the

Curve) score on how well a method can distinguish

between benign and adversarial examples; (3) True

Positive Rate (TPR) and False Positive Rate (FPR):

While TPR is the rate that an algorithm correctly

identifies adversarial examples, FPT is the rate that

such algorithm incorrectly detects benign inputs as

adversarial examples. We desire a high Model F1,

detect AUC, TPR, and a low FPR.

4.2 Results

Evaluation on Novice Attack. This section as-

sumes that the attacker does not know the existence

of trapdoors. Overall, DARCY significantly out-

performs other baselines, achieving detect AUC of

99% in most cases, with FPR less than 1% on av-

erage (Sec. A.2). Importantly, the clean prediction

performance of DARCY observes only 0.6% re-

duction on average. Compared to DARCY, SelfAtk

achieves similar performance in all except SST2

dataset with detection AUC only around 75% on

average (Fig. 3). This is because there are more

artifacts in the dataset that make the converged trig-

gers more diverse across several attack attempts.

We also experiment with selecting trapdoors ran-

domly. Fig. 4 shows that adaptive-based strategy

produces very stable results regardless training F

Method

+UE

+ScRNN

M +USE

R +SelfATK

+LID

textRNN

textCNN

F1 AUC FPR TPR F1 AUC FPR TPR

75.2 52.5 45.9 55.7 77.7 74.8 30.0 72.4

57.3 41.6 56.4

66.2 44.5 77.7

69.8 0.4 40.0

66.2 37.4 69.7

51.9 43.0 47.0

62.9 48.1 75.9

92.3 0.6 85.1

51.3 45.8 48.4

-

-

-

-

-

-

-

-

DARCY(1) 77.8 74.8 0.8 50.4 76.9 73.6 0.4 47.7

DARCY

78.1 92.3 2.9 87.6 77.4 91.2 3.2 85.5

Figure 5: Performance under Oracle Attack

+UE

+ScRNN

S +USE

J +SelfATK

+LID

89.4 34.5 62.5 43.1 89.6 59.9 44.2 64.7

55.0 53.6 62.9

72.7 38.8 83.1

72.8 0.5 46.0

67.5 32.0 67.1

57.6 51.1 65.7

70.7 41.4 81.6

80.7 8.0 69.3

50.7 54.3 55.7

-

-

-

-

-

-

-

-

DARCY(1) 89.4 71.7 0.6 43.9 88.5 70.8 4.9 46.6

DARCY

88.9 92.7 2.4 87.9 87.6 93.9 4.3 92.0

+UE

+ScRNN

S +USE

S +SelfATK

T +LID

79.0 50.6 48.8 52.5 77.7 77.7 26.3 74.2

56.1 19.1 31.2

55.2 55.4 70.4

61.8 0.2 23.8

64.0 18.8 46.9

53.8 19.2 26.8

60.8 50.1 72.2

66.1 3.7 35.9

49.9 62.2 61.9

-

-

-

-

-

-

-

-

DARCY(1) 82.9 69.7 0.2 39.6 77.3 59.3 0.9 19.6

DARCY

83.3 93.1 3.2 89.4 78.7 83.0 5.4 71.5

Figure 6: Performance under adaptive Attacks on

textCNN (top) and textRNN (bottom)

+UE

+ScRNN

A +USE

G +SelfATK

+LID

90.9 40.5 56.3 46.9 89.4 63.1 38.2 59.0

53.7 48.8 54.1

69.4 42.0 78.7

80.7 8.0 69.4

79.1 22.1 80.3

56.0 46.1 54.7

88.6 22.7 90.5

88.4 6.2 83.1

54.3 45.9 54.6

-

-

-

-

-

-

-

-

DARCY(1) 87.4 54.0 80.4 88.4 86.6 83.3 19.0 85.5

DARCY

89.7 95.2 9.3 99.8 88.6 92.6 14.7 99.9

All results are averaged over 15 runs on test sets

Table 4: Adversarial Detection Performance Averaged

Over All Target Labels under Advanced Attack

with a high ((cid:15) ← 1.0, “strong” trapdoors) or low

((cid:15) ← 0.1, “weak” trapdoors) trapdoor ratio (cid:15). How-

ever, trapdoors found by the random-based strategy

does not always guarantee successful learning of F

(low Model F1), especially in MR and SJ dataset

when training with a high trapdoor ratio (cid:15) (Fig. 41).

Thus, to have fair comparisons between the two

searching strategies, we only employ the “weak”

trapdoors strategy in all of the later experiments.

Evaluation on Advanced Attack. An ad-

vanced attacker tweaks the UniTrigger algorithm

to avoid selecting triggers associating with strong

local minimum on the loss landscape of F. To do

this, instead of always selecting the best tokens

from each iteration of beam-search (Sec. 2.1), an

expert can ignore the top P candidates and only

consider the rest to minimize the loss of F w.r.t

to the target label. Table 4 shows that DARCY

of 5 trapdoors outperforms other baselines when

P ← 20, achieving over 90% detect AUC in most

cases.

1AG dataset is omitted due to computational limit

Figure 7: Detection AUC v.s. # Query Attacks on

textCNN (top) and textRNN (bottom)

Evaluation on Oracle Attack. In this scenario,

the attacker has access to both F and the trapdoors

detection network G. With this assumption, the

attacker can incorporate G into the UniTrigger’s

learning process (Sec. 2.1) to generate triggers that

are undetectable by G. Fig. 5 shows the adversarial

detection results under the oracle attack. We ob-

serve that the detection performance of DARCY

significantly decreases regardless of when a sin-

gle or multiple trapdoors are used. Even though

increasing the number of trapdoors K : 1 → 5

lessens the effect on textCNN, oracle attack shows

that the access to G is the key to develop a robust

attack against honeypot-based defense algorithm.

Evaluation on Adaptive Attack. This section

assumes that attacker is aware of the existence of

trapdoors yet does not have access to G. Thus,

to attack F, the attacker adaptively replicates G

with a surrogate network G(cid:48), then generates trig-

gers that are undetectable by G(cid:48). To train G(cid:48), the

attacker can execute Q # of queries to generate

Figure 8: Detection TPR v.s. # Ignored Tokens

loss of G(cid:48) to find the optimal triggers. Overall, with

K ≤ 5, an advanced adaptive attacker can drop

TPR by as much as 20% when increases P :1 → 10

(Fig. 8). However, with K ≥ 10, DARCY be-

comes more robust against the advanced adaptive

attack. Fig. 8a also illustrates that adaptive-based

trapdoors searching is robust regardless of the #

of trapdoors K, while the random-based strategy

poorly performs when K is small (K < 5).

We further challenge DARCY by increasing up

to P = 30 (out of a maximum of 40 used by beam-

search). Fig. 9 shows that the more trapdoors

embedded into F, the more robust the DARCY

will become. While textCNN is more vulnerable

to advanced adaptive attacks than textRNN, using

30 trapdoors for each label will guarantee a robust

defense even under adaptive attacks.

Evaluation under Black Box Attack. Even

though UniTrigger is a white box attack, it can

also work in a black box setting via transferring

triggers S generated on a surrogate model F (cid:48) to

attack F. As several works (e.g., (Papernot et al.,

2017)) has proposed different methods to steal, i.e.,

replicate, F to create F (cid:48), we are instead interested

in examining if trapdoors injected in F (cid:48) can be

transferable to the F? To answer this question, we

use the model stealing method proposed by (Paper-

not et al., 2017) to replicate F (cid:48) using Dattack. Ta-

ble 7 (Appendix) shows that injected trapdoors are

transferable to a black box textCNN model to a cer-

tain degree on all datasets except SST. Since such

transferability greatly relies on the performance of

the model stealing technique and as well as the

tested datasets, future works are required to draw

any further conclusions.

5 Discussion

Case Study: Fake News Detection. UniTrigger

can help fool fake news detectors. We use one of

the latest public fake news datasets which include

over 4K news articles 2. A CNN-based detector

achieves 75% in accuracy on average on the test set.

2truthdiscoverykdd2020.github.io/

Figure 9: Detection AUC and Attack ACC v.s. # Ig-

nored Tokens on textCNN (top) and textRNN (bottom)

several triggers through F, and considers them as

potential trapdoors. Then, G can be trained on a

set of trapdoored-injected examples curated on the

Dattack set following Eq. (2) and (4).

Fig. 6 shows the relationship between # of trap-

doors K and DARCY’s performance given a fixed

of attack queries (Q = 10). An adaptive at-

tacker can drop mean TPR to nearly zero when

F is injected with only one trapdoor for each la-

bel (K = 1). However, when K ≥ 3, the TPR

quickly increases to over 80% and fully recovers at

above 98% when K ≥ 5. This confirms the robust-

ness aspect of DARCY against robust attacks as

described in Sec. 3.2. Moreover, detection TPR(s)

of adaptive-based and random-based strategies con-

verge as we increase the # of trapdoors. However,

Fig. 6 shows that adaptive searching results in a

much less percent of true trapdoors being revealed,

i.e., revealed ratio, by the attack on textCNN. More-

over, as Q increases, we expect the attacker will

gain more information from F, thus further drop

DARCY’s detect AUC. However, Fig. 7 shows that

DARCY is robust when Q increases, regardless of

the # of trapdoors. This is because universal trig-

gers usually converged to only a few true trapdoors

embedded in F even when initial tokens used by

UniTrigger are randomized.

Evaluation on Advanced Adaptive Attack.

An advanced adaptive attacker does not only repli-

cate G with G(cid:48), but also ignore top P tokens during

beam-search as in the advanced attack (Sec. 4.2)

to both maximize the loss of F and minimize the

Pruning %

MR

SJ

SST

AG

Parameters Analysis. We refer the readers to

F1 AUC F1 AUC F1 AUC F1 AUC

the Appendix for detailed analysis of γ, (cid:15).

20%

50%

64.9 99.3 80.0 99.2 37.3 68.2 17.1 98.5

51.3 91.9 82.6 99.4 66.6 50.3 11.9 87.3

6 Related Work

Table 5: Model F1 / Detect AUC of textCNN under

Trapdoor Removal using Model-Pruning.

UniTrigger is able to find a fixed 3-token trigger

to the end of any news articles of over 660 words

on average to drop its accuracy in predicting real

and fake news to only 5% and 16%, respectively.

Using Gunning Fog (GF) (Gunning et al., 1952)

score and a human study, we observe that the trig-

ger marginally reduces the readability of the news

articles (↑1 out of a 17-scale for GF, ↓0.23 out of

10-scale for human), especially to longer-length

articles (Appendix, Table 9). The human study in-

cludes 78 users on Amazon Mechanical Turk, each

is instructed to spend at least 1 minute reading a

news article and give a score from 1 to 10 on its

readability (Figure 10, Appendix). This shows that

UniTrigger is a very strong and practical attack.

However, by using DARCY of only 3 trapdoors we

are able to detect UniTrigger’s attacks up to 99% of

the time on averaged without assuming the triggers

are going to be appended (and not prepended) to

the target articles.

Trapdoor Detection and Removal. The attack-

ers may employ various backdoor detection tech-

niques (Wang et al., 2019b; Liu et al., 2019a; Qiao

et al., 2019) to detect if F contains trapdoors. How-

ever, these are built only for image domain do not

work well when the majority of all of the labels

have trapdoors (Shan et al., 2019) like in the case

of DARCY. Recently a few works propose to de-

tect backdoors in text. However, they either assume

the access to the training dataset (Chen and Dai,

2020), which is not always available, or not appli-

cable to trapdoors detection (Qi et al., 2020). The

attackers may also use model-pruning method to

remove installed trapdoors from F as suggested by

(Liu et al., 2018). However, by dropping up to 50%

of trapdoor-embedded F’s parameters with the

lowest L1-norm (Paganini and Forde, 2020), we

observe that F’s F1 significantly drops by 30.5%

on average. Except from SST dataset, however, the

detection AUC still maintains 93% on average.

Extension to other Models/Tasks. We observe

that DARCY also works well on transformer-

based models such as BERT (Devlin et al., 2019)

or RoBERTa (Liu et al., 2019b), each of which

achieves up to 99% in detection AUC on average.

Adversarial Text Detection. Adversarial detec-

tion on NLP is rather limited. Most of the cur-

rent detection-based adversarial text defense meth-

ods focus on detecting typos, misspellings (Gao

et al., 2018; Li et al., 2018; Pruthi et al., 2019)

or synonyms substitution (Wang et al., 2019c).

Though there are several uncertainty-based adver-

sarial detection methods (Smith and Gal, 2018;

Sheikholeslami et al., 2020; Pang et al., 2018) that

work well with computer vision, how effective they

are on the NLP domain remains an open question.

Honeypot-based Adversarial Detection (Shan

et al., 2019) adopts the “honeypot” concept to im-

age domain. While this method, denoted as GCEA,

creates trapdoors via randomization, DARCY gen-

erates trapdoors adaptively. Moreover, DARCY

only needs a single network G for adversarial detec-

tion. In contrast, GCEA requires a separate neural

signature (e.g., neural activation of the last layer)

for each trapdoor and compares all of these against

those of future inputs to detect harmful examples.

This then induces overhead calibrations to calcu-

late the best detection threshold for each trapdoor.

Furthermore, while (Shan et al., 2019) and (Carlini,

2020) show that true trapdoors can be revealed and

clustered by attackers after several queries on F,

this is not the case when we use DARCY to defend

against adaptive UniTrigger attack (Sec. 4.2). Re-

gardless of the initial tokens (e.g., “the the the”),

UniTrigger converges to a very small set of triggers

across multiple attacks regardless of # of injected

trapdoors. Investigation on whether this behavior

is generalized to other models and datasets is one

of our future directions.

7 Conclusion

This paper proposes DARCY, an algorithm that in-

jects multiple trapdoors, i.e., honeypot, adaptively

into a textual NN model to defend it against Uni-

Trigger’s adversarial texts. DARCY achieves as

high as 99% TPR and less than 1% FPR in most

cases across four public datasets. We also show

that DARCY with more than one trapdoor is robust

against even an advanced attacker. While DARCY

only focuses on defending against UniTrigger at-

tack, we hope to extend DARCY to safeguard other

NLP adversarial generators in the future.

References

Nicholas Carlini. 2020. A partial break of the hon-

eypots defense to catch adversarial attacks. arXiv

preprint arXiv:2009.10975.

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua,

Nicole Limtiaco, Rhomni St John, Noah Constant,

Mario Guajardo-Cespedes, Steve Yuan, Chris Tar,

arXiv

et al. 2018. Universal sentence encoder.

preprint arXiv:1803.11175.

Chuanshuai Chen and Jiazhu Dai. 2020. Mitigating

backdoor attacks in lstm-based text classification

systems by backdoor keyword identification. arXiv

preprint arXiv:2007.12070.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and

Kristina Toutanova. 2019. Bert: Pre-training of deep

bidirectional transformers for language understand-

ing. In NAACL-HLT (1), pages 4171–4186.

Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing

Dou....