Deep Learning for Natural Language Processing
Convolutional Neural Network (CNN)
Long Short Term Memory (LSTM)
Dr. Eng. Wael Ouarda
Assistant Professor, CRNS, Higher Education Ministry, Tunisia
Centre de Recherche en Numérique de Sfax , Route de Tunis km 10 , Sakiet Ezzit , 3021 Sfax – Tunisie
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Training phase
Image Database
Preprocessing
Features
Representation
CNN Architecture
Features
Classification
Dataset
Model F
Class 1
Class 2
Class 1
Class 2
Testing phase
Projection on the model
Unkonwn Image
Preprocessing
Features
Representation
F(X|X=”I”)=P1
F(X/X=”II”)=P2
max(P1,P2)
Unkonwn Vector
X
Wael Ouarda - CRNS
2
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Hubel and Wiesel Experiments (1962)
➔ Insert electrodes into specific parts of the visual
cortex of the cat;
➔ Measurement of activation when the cat saw
some basic shapes;
➔ The visual Cortex is the responsible of Perception;
➔ A cell of Neurons which are organized in Column
LeCun, Bottou, Bengio and Haffner (1998)
➔ Introduction of Convolutional Neural Network
(CNN);
➔ Inspiration from Visual Cortex
➔ Organization in Column of each Layer of the
Neural Network Architecture
Wael Ouarda - CRNS
3
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Convolutional Neural Network (CNN)
❏ A Feedforward Neural Network;
❏ Emulate the Visual Cortex in the Visual Perception Task;
❏ CNN have specific Layers that encodes the properties of Images
❏ Low Level Features;
❏ Middle Level Features;
❏ High Level Features.
❏ MLP Vs CNN
❏ MLP: Organization of Neurons into sequence of Layers
❏ CNN: Organization of Neurons into sequence of 3D-Layers (called Depth)
Wael Ouarda - CRNS
4
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Convolutional Neural Network (CNN) defines specific layers:
➔ Convolutional Layer (CL)
◆ Convolution Operator
◆ Linear Representation (Sum of Product)
➔ Pooling Layer (PL)
◆ Downsampling
Average Operator
●
● Max Operator
➔ ReLU Layer (RL)
◆ Help the optimization of the Gradient Descent
◆ To introduce the Non-Linear Representation
➔ Fully Connected Layer (FC)
◆ Emulate MLP Classifiers
Advertisement
◆ Classification Task
Wael Ouarda - CRNS
5
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Convolutional Layer (CL)
❏ Convolutional Layer
❏ based on Convolution Operator;
❏ The most important concept in Signal Processing;
❏ Construct the Output of any System
knowing its inputs and its Impulse Response.
❏ From 1D to 2D Convolution
❏ Convolution 1D for Signal Processing;
❏ Convolution 2D for Image Processing.
❏ Convolutional Kernel
❏ Used to be applied on Image;
❏ examples of kernel are used in Image processing
❏ Edge Detection;
❏ Blurring Image;
❏ Smoothing, etc.
Wael Ouarda - CRNS
6
Convolutional Neural Network (CNN) Architecture:
Convolutional Layer (CL)
Learning & Transfer Learning
*
1
0
2
0
1
0
-1
-2
-1
Kernel h
Input x
-13
-20
-17
Wael Ouarda - CRNS
7
Convolutional Neural Network (CNN) Architecture:
Convolutional Layer (CL)
Learning & Transfer Learning
*
1
0
2
0
1
0
-1
-2
-1
Input x
Kernel h
-13
-20
-17
-18
-24
-18
Wael Ouarda - CRNS
8
Convolutional Neural Network (CNN) Architecture:
Convolutional Layer (CL)
Learning & Transfer Learning
*
1
0
2
0
1
0
-1
-2
-1
Input x
Kernel h
-13
-20
Advertisement
-17
-18
-24
-18
13
20
17
Wael Ouarda - CRNS
9
Convolutional Neural Network (CNN) Architecture:
Convolutional Layer (CL)
Learning & Transfer Learning
Let fix our kernel of convolution h
Zero Padding
Sliding Window (Stride)
Wael Ouarda - CRNS
10
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Convolutional Layer (CL)
Parameters of CL
● Kernel size(K);
● Stride(S): Sliding Window (1 for CL and 1 for PL);
● Number of filters(F): Number of filters
● Zero Padding: Number of zeros to be add
Activity
Given I an input image 32x32x3
6 Kernel Filters of 5x5x3
Pad=0
Slide=1
The result will be an activation map 28x28x6.
Explain it!
Wael Ouarda - CRNS
11
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Convolutional Layer (CL)
Activity
Given I an input image 32x32x3
6 Kernel Filters of 5x5
Pad=0
Slide=1
The result will be an activation map 28x28x6.
Explain it!
= +
( + ∗ ) −
= +
+ ∗ −
= + =
Wael Ouarda - CRNS
12
Convolutional Neural Network (CNN) Architecture:
Pooling Layer (PL)
Learning & Transfer Learning
Spatial Pooling (subsampling or downsampling) reduces the dimensionality of each feature map but retains
the most important information. Spatial Pooling can be of different types: Max, Average, Sum, etc.
Wael Ouarda - CRNS
13
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
ReLU Layer (RL)
ReLU stands for Rectified Linear Unit and is a non-linear operation. Its output is given by:
Wael Ouarda - CRNS
14
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
FC Layer (FC)
eatures
eatures
Wael Ouarda - CRNS
15
Features Vector
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Conv2D,Flatten,MaxPooling2D
model = Sequential()
model.add(layers.Conv2D(256, (3, 3), activation='relu',input_shape=(227, 227, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(256, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
Advertisement
model.add(layers.Dense(256, activation='sigmoid'))
model.add(layers.Dense(1, activation='sigmoid'))
Input
image
227x227
Conv2D+relu
3X3X256
Padding=0
Stride=1
225
Max Pooling
2x2
256
112
112
225
Conv2D+relu
3X3X256
Padding=0
Stride=1
256
F
a
l
t
t
i
n
g
L
a
y
e
r
55
55
Max Pooling
2x2
110
256
256
110
16
256
55x55x256
Vertical
Horizontal
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
How to Design your Own CNN Architecture?
Wael Ouarda - CRNS
18
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
ImageNet Large Scale Visual Recognition Competition
Wael Ouarda - CRNS
19
Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Transfer Learning as Solution for lack of data!
Wael Ouarda - CRNS
20
6. Convolutional Neural Network (CNN) Architecture:
Learning & Transfer Learning
Deep Learning Frameworks
Wael Ouarda - CRNS
21
from keras import layers
from keras import models
from keras.layers import Dropout,MaxPooling2D,Dense,Flatten,Conv2D
model = models.Sequential()
model.add(layers.Conv2D(64, (3, 3), activation='relu',
input_shape=(227, 227, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dense(256, activation='sigmoid'))
model.add(layers.Dense(1, activation='sigmoid'))
= +
Advertisement
( + ∗ ) −
227,227,3
64x3x3
Stride = 1
Padding =0
64x225x225
MaxPooling
2x2
64x112x112
128x3x3
Stride = 1
Padding =0
128x110x110
MaxPooling
2x2
1
1
128x26x26
MaxPooling
2x2
128x53x53
256
Wael Ouarda - CRNS
128x27x27
128x3x3
Stride = 1
Padding =0
128x55x55
22
Thank you for your attention
Dr. Eng. Wael Ouarda
E-mail: [email protected]
Phone: +216 21 23 69 36
Web: http://www.crns.rnrt.tn/research-team/brain4ict
Wael Ouarda - CRNS
23
Long Short Term Memory (LSTM)
Recurrent Neural Network
A recurrent neural network can be thought of as multiple copies of the same
network, each passing a message to a successor.
Problem of Long Term Dependecy
Long Short Term Memory (LSTM)
Long Short Term Memory
Long Short Term Memory (LSTM)
Activation Functions
especially used for models where we have to predict the
probability as an output.
Since probability of anything exists only between the range of 0
and 1, sigmoid is the right choice
mainly used classification between two classes
Long Short Term Memory (LSTM)
LSTM Cell state & Gates
Cell State is the key of the LSTM
Horizontal Line over the Cell
Ability to remove or add information to the cell state, regulated by
structures called gates
Gates = a sigmoid neural net layer and a pointwise multiplication operation
Sigmoid is giving values within 0 and 1 to determine the quantity of
information to add to the state cell
cell state
Long Short Term Memory (LSTM)
Step One – Forget Gate Layer
To decide what information we’re going to throw away from the cell state
1 represents “completely keep this”
0 represents “completely get rid of this.”
Long Short Term Memory (LSTM)
Step Two – Inpu Gate Layer
Decide what new information we’re going to store in the cell state
Part 1: a sigmoid layer called the “input gate layer” decides which values we’ll update
Part 2: a tanh layer creates a vector of new candidate values that could be added to the state.
Long Short Term Memory (LSTM)
Step Three – Update Cell State
1. Multiply the old state by the forget Gate to forget what we don’t need from the previous state
2. Adding the new cell state new candidate Ct*input gates to construct a new state based on the new input
Long Short Term Memory (LSTM)
Step four – Output
Decide what we are going to output
output will be based on our cell state, but will be a filtered version