Defending Adversarial Attacks via Semantic Feature Manipulation

n morphs, and record the predicted label by a pre-trained classifier. The classification accuracy resistance is evaluated by the proportion

of unchanged predictions among morphs. The histogram is the average resistance on classification accuracy for 2000 clean and perturbed

instances derived from four adversarial attacks on MNIST and CelebA, respectively.

or adversarial. However,

the reconstruction error of the

perturbed images derived from oblivious attacks (such as

the CW attack) is very likely similar to normal images. The

threshold should be as low as possible to identify slightly

perturbed adversarial examples, considering that too low

would significantly misjudge normal examples. Besides, for

image transformation-based detection, a mixture of normal

and adversarial examples is required in the training set to

train the classifier. This causes the high computational cost

to generate adversarial examples and adversarial classifiers,

Latent Codes VaryingLatent Codes VaryingClassifierDetectorResistance Threshold Vector(e.g., θr=[50%,50%...] for all codes)Distance Threshold θdf9f13f16...Resistance Vectoron Classification CleanAcceptedSalvageableYNEncoderDecoderEncoderDecoderf9f13f16..................1/20=5%2/20=10%1/20=5%..................2277Clean7CleanAdversarialResistance Vector on Classification 20/20=100%20/20=100%20/20=100%...20/20=100%20/20=100%20/20=100%...Resistance Vector on Classification 20/20=100%20/20=100%20/20=100%...AdversarialRejectedYr1→r1→r2→r2→→Dis( ri, θr )≥θd→→Dis( ri, θr )≥θd→→ri ≥ θr , ∀ kri ≥ θr , ∀ kkkri ≥ θr , ∀ kkkNPurifierEncoderDecoderEncoderDecoder77Reconstructed7ReconstructedClean20/20=100%20/20=100%20/20=100%18/20=90%1/20=5%3/20=15%0/20=0%4/20=20%f9f13f16...f9f13f16...Resistance VectorVarying every latent code for 20 times on the same instanceAdversarial5/20=25%20/20=100%20/20=100%0/20=0%0/20=0%0/20=0%f2f3f5f2f3f5AdversarialClean7/7=100%0/7=0%f1f1CleanAdversarial7/7=100%0/7=0%f1f1CleanAdversarial0102030405060708090f3f9f13f16CleanFGSMCWDeepfoolAverage Resistance for Digits0102030405060708090100f1CleanFGSMCWDeepfoolAverage Resistance for Faceparticularly for more complex datasets and stronger attacks.

Further, the impact of external features transformation, such

as image rotation and shifting, is not consistent and general

for different instances. Besides, the background of an image

adds a large number of extra features to the object, which

is also sensitive to the external transformation.

Completeness. Generally, there are two types of semantic

features: class-shared (such as the thickness of the hand-

written digits, the facial expression of the face images) and

class-unique (such as different handwriting style for each

digit or the identification of face). However, the reformer of

MagNet and the generator of FBGAN and Defense-GAN can

only reconstruct/generate the purified instance using some

commonly shared feature. It leads to loss of class-unique

features, for example, different writing styles exclusive to

the digit ”2” (such as a flat stroke or across loop bottom);

such features become inactive for some classification tasks,

e.g., writer identification or face recognition. As the high

dimension space input, e.g., image, always lies in a complex

manifold, the underlying data distribution could be very

complex. The purifier is used to build complex enough

models to capture the true posterior by utilizing both class-

shared and class-unique features.

image is generally resistant

In this paper, we propose an adversarial example detec-

tion and purification method, named Feature-Manipulation

defense (FM-Defense), to address these two concerns. It

can effectively defeat the state-of-the-art adversarial attacks,

including CW attack. The intuition is that the classification

result of a normal

to non-

significant intrinsic feature changes, e.g., varying thickness

of handwritten digits or the facial expression. Namely, the

classification results of a normal digit and its morphs, de-

rived by varying thickness, are very likely to be stable, since

the significant features are retained. In contrast, adversarial

examples are sensitive to such changes. The reason is that

the unstructured perturbation is designed for a single image,

and may cause various impacts on its morphs due to lack

of transferability. Figure 1 demonstrates our intuition using

a handwritten digit, trouser shape and face image.

The key point

to implement our intuition is how to

manipulate the intrinsic feature. Consequently, a one-off

combo-variational autoencoder (combo-VAE) is applied to

learn disentangled low-dimensional latent codes, i.e., one

latent code only affects one semantic feature. The learned

latent codes are disentangled, easy to control, and composed

of abundant internal semantic features, instead of external

features such as image rotation and shifting. The resistance

of classification accuracy when manipulating the intrinsic

features via disentangled latent codes is used to detect suspi-

cious inputs. As demonstrated in the histograms of Figure 1,

the classification accuracy change of clean instances is more

consistent than that of adversarial instances. Hence, a simple

threshold of classification accuracy resistance can be set to

easily distinguish normal and adversarial images instead of

training the adversarial classifier. Further, the combo-VAE

is also applied to purify the suspicious instances close to

the manifold of normal examples by reconstructing using

both class-shared and class-unique features to move them

towards the manifold. To the best of our knowledge, the FM-

Defense is the first attempt to apply disentangled learning

for effective defense against oblivious adversarial attacks via

both detection and purification, with good interpretability,

feasibility, and completeness.

Our contributions are summarized as follows. We first

present a key intuition that adversarial examples are gener-

ally more sensitive to intrinsic feature changes than normal

images. Based on this intuition, we then propose feature

manipulation-based adversarial example detection and pu-

rification method, FM-Defense. We use a combo-VAE to

manipulate the feature in an easy and interpretable manner.

Besides detection, the combo-VAE is used to purify the

suspicious inputs by reconstructing images based on both

class-unique and class-shared components. It can improve

the completeness of the reconstructed instance for purifi-

cation. We implement and evaluate the FM-Defense on

three image datasets, MNIST, FMNIST and CelebA, which

shows the superior performance in defending against various

adversarial attacks.

II. BACKGROUND AND RELATED WORK

A. Autoencoders and β-VAE

Autoencoders (AEs) are common deep models in unsuper-

vised learning [10]. They aim to represent high-dimensional

data through the low-dimensional latent layer, a.k.a. bottle-

neck vector or code. Architecturally, AEs consist of two

parts, the encoder and decoder. The encoder part takes the

input x ∈ Rd and maps it to z (the latent variable of the

bottleneck vector). The decoder tries to reconstruct the input

data from z. The training process of autoencoders is to

minimize the reconstruction error. Formally, we can define

the encoder and the decoder as transitions τ1 and τ2:

τ1(X) → Z

τ2(Z) → ˆX

(cid:13)

(cid:13)

2

(cid:13)X − ˆX

(cid:13)

(cid:13)

(cid:13)

τ1, τ2 = argmin

τ1,τ2

(1)

The VAEs model shares the same structure with the au-

toencoders, but is based on an assumption that the latent

variables follow some kind of distribution, such as Gaussian

or uniform distribution. It uses variational inference for the

learning of the latent variables. In VAEs the hypothesis is

that the data is generated by a directed graphical model

p(x|z) and the encoder is to learn an approximation qφ(z|x)

to the posterior distribution pθ(z|x). The VAE optimizes the

variational lower bound:

L(θ, φ; x) = KL(qφ(z|x)||pθ(z)) − Eqφ(z|x)[logpθ(x|z)]

(2)

The left part is the regularization term to match the posterior

of z conditional on x, i.e., qφ(z|x), to a target distribution

pθ(z) by the KL divergence. The right part denotes the

reconstruction loss for a specific sample x. In a training

batch, the loss can be averaged as:

LV AE = Epdata(x)[L(θ, φ; x)]

= Epdata(x)[KL(qφ(z|x)||pθ(z))]−

Epdata(x)[Eqφ(z|x)[logpθ(x|z)]]

(3)

β-VAE is a modification of the VAE framework that

introduces an adjustable hyperparameter β to the original

VAE objective:

L = Eqφ(logpθ(x|z)) − βDKL(qφ(z|x)||pθ(z))

(4)

Well chosen values of β (usually β > 1) result in more

disentangled latent representations z. When β = 1, the β-

VAE becomes equivalent to the original VAE framework. It

was suggested that the stronger pressure for the posterior

qφ(z|x), to match the factorized unit Gaussian prior p(z)

introduced by the β-VAE objective, puts extra constraints

on the implicit capacity of the latent bottleneck z. Higher

values of β necessary to encourage disentangling often lead

to a trade-off between the fidelity of β-VAE reconstructions

and the disentangled nature of its latent code z (see Fig. 6

in [15]). This is due to the loss of information as it passes

through the restricted capacity latent bottleneck z.

B. Adversarial attacks

that

Evasion attacks have long been studied on machine learn-

ing classifiers [11], [12], and are practical against many types

of models [13]. These evasion attacks over neural networks

are referred to as adversarial examples [14]. Namely, for a

given input sample x, the adversarial example is a sample

x(cid:48)

is similar to x (according to particular measure

metrics) but so that the classifier’s decision C(x) (cid:54)= C(x(cid:48))

[13]. A classifier can misclassify an adversarial example

for two reasons. (1) The adversarial example is far from

the boundary of the manifold of the task. For example, the

task is a handwritten digit classification, and the adversarial

example is an image containing no digit, but the classier

has no option to reject this example and is forced to output

a class label. (2) The adversarial example is close to the

boundary of the manifold. If the classier poorly generalizes

the manifold in the vicinity of the adversarial example, then

misclassification occurs.

Let U be the set of all instances in the sample space.

A classification function is denoted by C, which outputs

for each instance x ∈ U a predicted class C(x) = y. Let

Y = {y1, · · · , ym} denote the set of classes for a certain

classification task. Each classification function assumes a

data generation process that produces each instance x ∈ U

with probability p(x). Let N be a manifold that consists of

instances that act naturally with regard to a certain classifi-

cation task, following a data generation process. N can be

approximated by a set of natural instances for a classification

task [6], e.g., MNIST. The goal of the adversarial example

is to find certain perturbation on x to generate adversarial

example x∗ ∈ U \ N that fools a specific C to misclassify,

i.e. C(x∗) (cid:54)= C(x).

The adversary is assumed to have the knowledge of the

original classifier but is not aware of the detector and purifier.

Therefore, the goal of the adversary is only to fool the

unsecured classifier.

C. Adversarial defenses

Defense on neural networks is much harder compared

with attacks. We summarize some ideas of current ap-

proaches to defense and compare them to our work.

1) Adversarial Training: One idea of defending against

adversarial examples is to train a better classifier [15]. An

intuitive way to build a robust classifier is to include adver-

sarial information in the training process, which we refer to

as adversarial training. For example, one may use a mixture

of normal and adversarial examples in the training set for

data augmentation [14], or mix the adversarial objective with

the classification objective as regularizer [16]. Although this

idea is promising, it is hard to reason about what attacks

to train on and how important the adversarial component

should be. Currently, these questions are still unanswered.

2) Defensive Distillation: Defensive distillation [17]

trains the classifier in a certain way such that it is nearly

impossible for gradient-based attacks to generate adversarial

examples directly on the network. Defensive distillation

leverages distillation training techniques [18] and hides the

gradient between the pre-softmax layer (logits) and softmax

outputs. However, [19] showed that it is easy to bypass the

defense by adopting one of the three following strategies:

(1) choose a more proper loss function (2) calculate gradient

directly from pre-softmax layer instead of from post-softmax

layer (3) attack an easy-to-attack network first and then

transfer to the distilled network.

3) Detecting Adversarial Examples: The detection-based

defense against adversarial examples for a classifier C aims

to establish a detector dC : U → Y ∪ {J}. J is the judgment

that the input is unlikely from the manifold of the normal

instances. Further, the purification-based defense is to build

a purifier p : U \ N → N to reconstruct suspicious instances

with small distortion only using some essential features, in

order to move adversarial examples towards the manifold of

normal examples. The defense aims to increase the accuracy

of the classifier with the presence of adversarial examples

by (1) detecting the input as an adversarial example or

a normal image while rejecting suspicious instances with

huge distortion; (2) purifying suspicious instances with small

distortion by reconstruction. One strong defense is to detect

adversarial examples with hand-crafted statistical features

[3] or separate classification networks [4]. A representative

work of this idea is [4]. For each attack generating method

considered, it constructed a DNN classifier (detector) to tell

whether an input is normal or adversarial. The detector was

directly trained on both normal and adversarial examples.

The detector showed good performance when the training

and testing attack examples were generated from the same

process, and the perturbation was large enough, but it did

not generalize well across different attack parameters and

attack generation processes.

III. DEFENSE VIA SEMANTIC FEATURE MANIPULATION

A. FM-Defense overview

We propose Feature Manipulation based defense (FM-

Defense), a framework for defending adversarial examples

via detecting and purifying.

Figure 2: Scheme of FM-Defense.

Three key challenges need to be addressed by FM-

Defense: (1) How to make the feature manipulation feasible

in an interpretable manner? (2) How to improve the dis-

crimination ability of the detector to recognize adversarial

examples by manipulating features efficiently? (3) How

to enhance the quality and completeness of the purified

instances by reconstruction? To address these three issues,

FM-Defense is composed of three components:

(1) Representation with improved disentanglement. We

initially train a disentangle representation model, combo-

VAE, enhanced from β-VAE [20], on a clean dataset. The

objective here is to make disentangled latent codes that have

semantic meaning so that they are easy to be controlled.

Such models consist of two components: the encoder E :

X → Z takes the input x ∈ Rd and maps it to z (the latent

variable of the bottleneck vector). The decoder D : Z → X

tries to reconstruct the input data from z. The encoder can

map high-dimensional input instance x to disentangled low-

dimensional latent codes z, i.e., the one latent code can

only control one certain feature. The decoder is used to

reconstruct the input from the low-dimensional latent code z.

For simple images, e.g. 28×28 handwritten digits, the latent

code can be a m-dimensional vector. For complex images,

e.g. 128 × 128 face images, the latent codes are m channels

of n × n feature maps. We find that some n × n-dimensions

feature map can reveal some disentangled semantic features

as well. Therefore, each channel of n × n feature map can

be considered as one latent factor used for manipulation for

simplicity, namely, each element of the n × n feature map is

simultaneously changed at the same scale. The combo-VAE

to extract the disentangled latent codes for each instance so

that it is feasible to select and manipulate a number of codes

that reveal the desired semantic features, e.g. thickness of the

digit. The feasibility of feature manipulation is related to

the disentanglement level. Therefore, strategies are used to

improve the disentanglement, as described in the following

sections.

(2) Detector with fine discrimination ability. Given an

instance, we first vary a latent code i for T times to

obtain n morphs reconstructed by the decoder. We then

record the ratio of unchanged classification prediction by

applying a certain classifier (to be protected) on these n

morphs compared with the original prediction. The ratio

is used as a resistance indicator r(i) for code i. At one

time, we change one of m selected latent codes in turn

and obtain a m-dimensional resistance vector −→r for each

instance. We find the resistance ability of normal instances

is significantly better than that of adversarial ones, as shown

in the histogram of Figure 1. Therefore, a m-dimensional

−→

θr for all selected m latent codes

threshold configuration

can be decided on the normal instance to distinguish normal

and adversarial instances. An instance, that meets r(i) >

r , ∀ r(i) ∈ −→r , will be recognized as normal. Otherwise,

θ(i)

it is recognized as suspicious.

(3) Purifier with comprehensive reconstruction ability. It

is feasible to decide a threshold that can achieve nearly 100%

adversarial detection accuracy (True Positive). However, this

will cause a large number of normal instances to be im-

properly recognized as adversarial, i.e., high False Positive

ratio. We assume there exists a manifold of resistance on

clean instances. Therefore, we use another threshold θd over

the distance Dis(.) between the resistance vector of an

−→

instance −→r and the resistance threshold vector

θr , to build a

salvageable set consists of suspicious instances close to the

manifold of normal instances.

Dis(−→r ,

−→

θr ) =

(cid:88)

|r(i) − θ(i)

r |, ∀r(i) < θ(i)

r

(5)

As illustrated in Figure 3, if a suspicious instance from

the detection has a distance of more than θd, then it will

be rejected as adversarial. Otherwise, it is incorporated in a

salvageable set that will be reconstructed by a combo-VAE

Z_uxZ_sDGMMEncoder_SDecoderL_GMCµ1,µ2,...,µk Σ1,Σ2,...,Σk L_KLTC DiscriminatorL_TCL_DL_RPurifier uses Z_s+Z_uPurifier uses Z_s+Z_uDetector only uses Z_sDetector only uses Z_sx'x'the discriminator-based approximation as follows:

T C(z) ≈ Eq(z)[log

D(z)

1 − D(z)

]

(7)

The objective of combo-VAE is augmented with a TC

[21] term to encourage independence in the latent factor

distribution as follows:

Eqφ(z|x(i))[logpθ(x(i)|z) − LKL(qφ(z|x(i))||p(z))] − γLT C

(8)

Note that this is also a lower bound on the marginal log-

likelihood Ep(x)[logp(x)]. The first part reveals the recon-

struction error, denoted by LR, evaluating whether the latent

bottleneck vector z is informative enough to recover the

original instance. LR can be defined as the l2 loss between

the original instance and the reconstructed instance. The

second part is a regularization term, denoted by LKL, to

push qφ(z|x) to match the prior distribution p(z). The third

is the TC term, denoted by LT C,

part

to measure the

dependence for multiple random variables.

As

shown in Figure 2,

the parameter φ of en-

coder qφ(z|x) is then trained by LKL, LR and LT C

in terms of −∇φ(LKL + LR + γLT C). The param-

is updated in terms of −∇θ(LR).

eter θ of decoder

The parameter υ of TC-discriminator

is updated in

i∈B log(Dυ(z(i)) +

terms of −∇υ(LT ),

(cid:80)

i∈B(cid:48) log(1−Dυ(permutedim(z(cid:48)(i)))]. Here, the permute-

dim function is to random permutate on a sample in the batch

for each dimension of its z, similar to [22].

2|B| [(cid:80)

i.e. −∇υ

1

C. Detector with fine discrimination

The indicator for adversarial detection should easily dif-

ferentiate normal and adversarial instances, be feasible and

stable to conduct, and attack-agnostic. The classification

accuracy resistance over the morphs, derived from feature

manipulation by changing a certain latent code, can meet

these criteria. The discrimination ability of the detector

depends on the ability to reduce the false-positive ratio

instances to be recognized as adversarial) and

(normal

the naturality of the morphs. Consequently, we apply two

strategies: normal value range selection and natural morph

generation.

Natural morph generation. The initial step is to find

the normal value range of each code on the clean valida-

tion set, then the morphs are produced via manipulating

each code within its normal value range. As the latent

codes are disentangled, independent (all from N (0, 1)) and

have semantic meaning, some latent codes that reveal non-

significant intrinsic features (e.g., thickness for handwritten

digits) will be selected and their normal range can be decided

empirically in an interpretable manner for human on a vali-

dation set. To obtain the morphs by feature manipulation, we

can incrementally add/reduce a fixed value on the original

learned latent codes within the normal range. However, the

Figure 3: Threshold settings.

based purifier. The purifier moves suspicious examples in

the set towards the manifold of normal examples to correctly

classify adversarial examples with small distortion or clean

instances improperly recognized as adversarial. Namely, the

purifier reconstructs close approximation on the normal

manifold before feeding to the target classifiers. To improve

the quality of reconstructed instance, we enhance the combo-

VAE to absorb both significant class-unique features and

class-shared features. Details of these components are given

in the following sections.

B. Representation with improved disentanglement

VAE-based autoencoders and their variations are com-

monly applied for disentanglement learning. Specifically,

the encoder E, parameterized by qφ(z|x),

is trained to

convert high-dimensional data x into the latent representa-

tion bottleneck vector z in the latent space that follows a

specific Gaussian distribution p(z) ∼ N (0, 1). The decoder

pθ(x|z) is trained to reconstruct the latent vector z to x. The

encoder and decoder are trained simultaneously based on

the negative reconstruction error and the regularization term,

i.e., Kullback-Leibler (KL) divergence between qφ(z|x) and

p(z). The regularization term is used to regularize the

distribution qφ(z|x) to be Gaussian distribution whose mean

µ and diagonal covariance (cid:80) are the output of the encoder.

We apply a combo-VAE to get good disentanglement in

z by improving the inner-independence of latent codes.

Specifically, Total Correlation (TC) [21] is used to encourage

independence in the latent vector z, as given in Equation 2.

T C(z) = KL(q(z)||¯q(z)) = Eq(z)[log

q(z)

¯q(z)

]

(6)

As TC is hard to obtain, the approximate tricks used in [22]

is applied to estimate TC. Specifically, a discriminator Dtc

is applied to classify between samples from q(z) and ¯q(z).

Thus learning to approximate the density ratio is needed for

estimating TC [22]. Dtc, parameterized by υ, is trained with

other components jointly. Thus, the TC term is replaced by

θrθdAdversarial rNormalNormalAdversarialNormal rSalvageable rf2f3f4f5f6f7f8f9f10f1→DistanceSalvageable→→→123modified latent vector maybe not be on the manifold of

normal

instance

instances. If that happens, an unnatural

will be reconstructed by the decoder. Hence, we conduct

an iterative stochastic search to make the morphs on the

manifold by adding natural noise. Specifically, we increase

the search range by ∆r within which the perturbation for a

certain latent code ∆zi is randomly sampled (B samples for

each iteration) until we produce N natural latent code with

the value in the normal value range to reconstruct N natural

morphs. We then evaluate the resistance on classification

for this latent code using the targeted classifier. Iterative,

we can get a m-dimensional resistance evaluation vector for

each instance.

Threshold selection strategies. Given a targeted classifier,

we decide a resistance threshold for each latent code on a

validation set containing only clean instances. The threshold

of resistance is decided for each factor so that the false-

positive rate on the validation set is below a pre-defined

value 1 − ρ (i.e. more than ρ% clean instance are correctly

recognized). Therefore, we can decide a unified and fixed

resistance threshold for all

latent codes or formulate a

specific threshold for each latent code in terms of ρ.

D. Purifier with comprehensive reconstruction

We assume that suspicious instances with a small distance

between their resistance vector and the resistance threshold

vector can be considered as close to the manifold of the nor-

mal instances. Namely, the distance is under the θd, which

is set as the η% fractile on clean validation data. Therefore,

a VAE-based purifier is used to reconstruct and move them

towards the manifold of normal examples. In regular VAEs,

the prior over the latent variables is commonly an isotropic

Gaussian, resulting in limited representation because the

learned representation can only be unimodal and does not

allow for more complex representations [23]. Consequently,

the regular VAE-based disentangled learning generally learn

some class-shared information and with some essential class-

unique information lost. This limitation causes only the

suspicious instances with tiny perturbation, while suspicious

instances with larger perturbation, e.g. derived from FGSM

attack in Figure 6, can not be reconstructed correctly. To

address the completeness and accuracy of the VAE-based

purifier, we enhance the combo-VAE by incorporating class

relevant conditional information to guide the reconstruction.

Our latent codes consist of two components: class-unique

representation, zu (e.g., important features unique to each

digit), and commonly shared across all classes, zs (e.g.,

the thickness of handwriting digits). As shown in Figure 2,

the enhanced combo-VAE has a similar scheme to a VAE,

but instead of using exclusively the same data for the input

and output of the network, we use class-unique additional

information as an extra input to the decoder.

Specifically, we assume the observed instances are derived

from a mixture of Gaussians, i.e. the inference of the class

of an instance is equivalent to inferring which mode of

the latent codes zu of the data point was generated from.

Namely, we use a mixture of Gaussians as our prior for

the zu, used as conditional information for training decoder

of VAE. For each class label c, we assume it has K

features such as different writing styles for handwritten

digits, namely K-dimensional zu. Therefore, we first train a

deep neural network using Gaussian Mixture loss (DGMM)

that maps input x to zu that is learned with the supervision

of the categorical class label c. zu reveals K features for the

label of a given input instance. The DGMM is solely trained

using the clean labeled instances, and the output is a logistic

regression on the latent representation of K features, as a

classification task.

Each feature z(k)

u

follows a mixture of K Gaussian

distribution with learned mean µc and covariance Σc for

each class c, given by neural networks of DGMM with

parameters κµ and κΣ respectively.

p(z(k)

u ) =

C

(cid:88)

c

N(c; µc, Σc)p(c)

(9)

Here, p(c) is the prior probability of class c. The loss

of DGMM is calculated as the cross-entropy between the

posterior probability q(c|zu) and the corresponding one-

hot class labels, denoted Lcls, combined with a likelihood

regularization term to force the training samples to obey the

assumed GM distribution, denoted Llkd [23], [24]. Lcls can

let zu contain as much label information as possible, as the

MI between zu and class c are added to the maximization

objective function.

Lcls = −Eqκ(zu|x)

(cid:88)

c

I(c = y)log q(c|zu)

= −log

N (zu; µy, Σy)p(y)

k N (zu; µk, Σk)p(k)

(cid:80)

(10)

Llkd is applied for measuring to what extent the training

samples fit the assumed distribution, which can be simplified

as Equation 7 when p(c) is simply set to 1/C for all classes.

The Llkd for a given class c is given as follows:

Llkd = −logN (zu; µc, Σc)

Consequently, the loss for DGM M is

LGM = Lcls + λlkdLlkd,

(11)

(12)

where λ is a non-negative weighting coefficient.

In addition, an encoder Es is trained to map input x

to zs where each code is forced to follow the standard

Gaussian N (0, I), implemented completely by the qφ(z|x)

in Section Representation with improved disentanglement.

The input of a pre-trained DGMM can a given instance x

without a label, since DGMM will output the K-dimensional

feature vector of the most likely class of x. The latent

codes zs and K-dimensional zu derived from the pre-trained

DGMM are then simply concatenated together to a decoder

to reconstruct the input x. The loss of decoder is used

to measure how probable it is to generate x by using the

distribution p(x|zu, zs), that is, is a distance between x and

reconstructed x.

The training of combo-VAE is two-stages. Initially, the

DGMM (modeled by qκ) is updated using LGM to learn

mean µc and covariance Σc of the prior p(zu|c), encouraging

zu to be label depe...