Transfer Learning

Dr. Haithem Hermessi

•

Sr. Com puter Vision Engineer @ SCY L L A

AI research Scientist @ LIMTIC - University of Tunis El Manar

[email protected]

Outline

• Introduction

• Transfer Learning

• Deep Transfer Learning

• Types of Deep Transfer Learning

• Implementation

• Building a Deep Transfer Learning Model with Keras in Python

Transfer Learning

• Transfer learning aims to leverage the learned knowledge from a resource-rich

domain/task to help learning a task with not sufficient training data.

• Sometimes referred as domain adaptation

• The resource-rich domain is known as the source and

the low-resource task is known as the target.

• Transfer learning works the best if the model features

learned from the source task are general (i.e., domain-

independent).

Transfer Learning Examples in Practice

• Image Processing: Learning an image recognition system on cats/dogs

and use it for cars/trains recognition.

• Which dataset has more labeled images?

• Do images in these two domains share common traits?

• Sentiment Analysis: Learning a sentiment analysis system on

Amazon’s laptops review and apply it to digital camera review.

• Should we manually label many camera reviews from scratch?

• What are source and target tasks/domain in the above examples?

Transfer Learning: Definition

• Let 𝒟𝑠 and 𝒟𝑡 denote the source and target domains, respectively.

• A domain contains the feature space.

• Also let 𝒯𝑠 and 𝒯𝑡 be the source and target tasks, respectively.

• Transfer learning aims to help improve the learning of the target

in 𝐷𝑡 using the knowledge in 𝐷𝑠 and 𝒯𝑠,

predictive function 𝑓𝑇 .

where 𝐷𝑠 ≠ 𝐷𝑡 and/or 𝒯𝑠 ≠ 𝒯𝑡.

Deep Transfer Learning

• The recent progress in deep learning has facilitated transfer learning

mainly because of two reasons:

1. Networks can be pre-trained on one domain and be tuned on another

domain.

2. Network weights can be shared among different tasks.

• A transfer learning task 𝒟𝑠, 𝒯𝑠,𝒟𝑡, 𝒯𝑡, 𝑓𝑇 .

is a deep transfer

is a non-linear function defined by a deep

learning task where 𝑓𝑇 .

neural network.

Deep Transfer Learning: How to proceed?

• The most common incarnation of transfer learning in the context of deep learning

is the following workflow:

• Take layers from a previously trained model.

• Freeze them, so as to avoid destroying any of the information they contain during

future training rounds.

• Add some new, trainable layers on top of the frozen layers. They will learn to turn

the old features into predictions on a new dataset.

• Train the new layers on your dataset.

Implementing Model Sharing-Based Deep

Transfer Learning

Python, TensorFlow, Keras

Deep Transfer Learning Implementation

• Prerequisites:

• Python 3.5+ (https://www.python.org/)

• TensorFlow (https://www.tensorflow.org/)

• Keras (https://keras.io/)

• A high-level library on top of TensorFlow, CNTK, or Theano.

• Recommended:

• NumPy

• Scikit-Learn

• NLTK

• SciPy

Implementation

• We build a Model Sharing-Based Deep Transfer Learning in Image

processing which uses CNN as the core neural network model.

• We use VGG-19, a pre-trained CNN on more than a million images from

ImageNet.

• Review the CNN tutorial to refresh your memory.

• VGG-19 pre-trained model is available as a built-in model in Keras.

Implementation – Determining the Libraries

from keras import applications

Contains VGG trained model in Keras

from keras.preprocessing.image import

ImageDataGenerator

Keras Builtin library for preprocessing images

from keras import optimizers

Contains different loss Functions used for BackProp

from keras.models import Sequential, Model

from keras.layers import Dropout, Flatten,

Dense, GlobalAveragePooling2D

Contains different type of layers

from keras import backend as k

Implementation – Loading the imagenet

img_width, img_height = 256, 256

train_data_dir = "data/train"

validation_data_dir = "data/val"

nb_train_samples = 4125

nb_validation_samples = 466

batch_size = 16

epochs = 50

Defining model training variables

model = applications.VGG19(weights = "imagenet", include_top=False,

input_shape = (img_width, img_height, 3))

Loading the pre-trained model as feature

extractor without including the top

classification layer.

Implementation – Feature Extractor

for layer in model.layers[:5]:

layer.trainable = False

Freeze the first 5 layers of the model (feature

extractor part of the pre-trained model)

#Adding custom layers

x = model.output

x = Flatten()(x)

x = Dense(1024, activation="relu")(x)

x = Dropout(0.5)(x)

x = Dense(1024, activation="relu")(x)

Adding custom layers that can be

updated.

predictions = Dense(16, activation="softmax")(x)

Adding the classification layer

Determining the model’s input and output

model_final = Model(input = model.input, output = predictions)

Compile the model

model_final.compile(loss = "categorical_crossentropy", optimizer =

optimizers.SGD(lr=0.0001, momentum=0.9), metrics=["accuracy"])

Implementation – Creating Training and Testing sets

train_datagen = ImageDataGenerator(

rescale = 1./255,horizontal_flip = True,fill_mode = "nearest",

zoom_range = 0.3,width_shift_range = 0.3,height_shift_range=0.3,

rotation_range=30)

test_datagen = ImageDataGenerator(

rescale = 1./255,horizontal_flip = True,fill_mode = "nearest",

zoom_range = 0.3,width_shift_range = 0.3,height_shift_range=0.3,

rotation_range=30)

train_generator = train_datagen.flow_from_directory(

train_data_dir,target_size = (img_height, img_width),

batch_size = batch_size, class_mode = "categorical")

validation_generator = test_datagen.flow_from_directory(

validation_data_dir,

target_size = (img_height, img_width),

class_mode = "categorical")

Pre-process train and

test data

Initiate the train and

test generators

Implementation – Training the model

Train the model

model_final.fit_generator(

train_generator,

samples_per_epoch = nb_train_samples,

epochs = epochs,

validation_data = validation_generator,

nb_val_samples = nb_validation_samples)

Train the whole model with

pre-defined training parameters.