Multi Layer Perceptron
Amine Kerkeni
Head of Engineering, InstaDeep
Google Developer Expert in Machine Learning
T
h
e
P
e
r
c
e
p
t
r
o
n
The Forward Propagation
The Forward Propagation
The Forward Propagation
The Forward Propagation
Activations
Activations
Activation functions main role is to introduce non-linearity
Publicité
A numerical example
A numerical example
A
N
e
u
r
a
l
N
e
t
w
o
r
k
w
i
t
h
P
e
r
c
e
Publicité
p
t
r
o
n
A Simple Perceptron
Multi Output Perceptron
A Neural Network
A Neural Network
A Deep Neural Network
T
h
e
L
o
s
s
Defining the Loss
The loss is the error between our prediction and the ground truth
Defining the Loss
The empirical loss is the error measured on the whole dataset
Binary Cross Entropy Loss
Used for models that output predictions between 0 and 1.
Mean Squared Error Loss
Used for regression models that output continuous real numbers.
Publicité
T
r
a
i
n
i
n
g
Gradient Descent
Gradient Descent
● Deep learning datasets are usually huge: 106∼108 training
examples. In consequence, computing the gradient on the full
training dataset can be extremely costly
● To tackle this, a solution is the compute the gradient of a
minibatch of samples and perform an update step with it
● This gradient is an unbiased estimation of the full gradient if
the minibatch sampling is performed randomly
Stochastic Gradient Descent
The Learning Rate
SGD with Momentum
Loss Optimization
Backpropagation
Backpropagation
Backpropagation
Backpropagation
Publicité
T
r
a
i
n
i
n
g
P
r
o
b
l
e
m
s
Overfitting
Dropout
Dropout is to randomly set some activations to 0 during training.
It forces the network to avoid relying too much on a single
activation.
Early Stopping