Deep Learning
M o t i v a t i o n a n d E v o l u t i o n
Dr. Haithem Hermessi
•
•
Sr. Compu te r Vision Engine e r @ SCYLLA
AI research Scientist @ LIMTIC - University of Tunis El Manar
Inspiration for Deep Learning: The Brain!
McCulloch & Pitts (1943): networks of binary neurons can do logic
Donald Hebb (1947): Hebbian synaptic plasticity
Norbert Wiener (1948): cybernetics, optimal filter,
feedback, autopoïesis, auto-organization.
Frank Rosenblatt (1957): Perceptron
Hubel & Wiesel (1960s): visual cortex architecture
2
Supervised Learning
Training a machine by showing examples instead of programming it
When the output is wrong, tweak the parameters of the machine
Works well for:
Speech→words
Image→categories
Portrait→ name
Photo→caption
Text→topic
….
C AR
PLAN E
3
Supervised Learning goes back to the Perceptron & Adaline
The McCulloch-Pitts Binary Neuron
Perceptron: weights are motorized potentiometers
Adaline: Weights are electrochemical “memistors”
N
y= sign(∑ W i X i+ b)
i = 1
https://youtu.be/X1G2g3SiCwU
4
The Standard Paradigm of Pattern Recognition
...and “traditional” Machine Learning
Feature
Extractor
Trainable
C las sifier
Hand engineered
Trainable
5
Multilayer Neural Nets and Deep Learning
Traditional Machine Learning
Feature
Extractor
Hand engineered
Trainable
Classifier
Trainable
Deep Learning
Trainable
Low-Level
Features
Mid-Level
High-Level
Trainable
Features
Features
Classifier
6
(Deep) Multi-Layer Neural Nets
Multiple Layers of simple units
Each units computes a weighted sum of its inputs
Weighted sum is passed through a non-linear function
The learning algorithm changes the weights
R e L U ( x )= m a x ( x , 0 )
Ceci est une voiture
Weig ht
matrix
Hidden
Layer
7
Supervised Machine Learning = Function Optimization
Function with
adjustable parameters
Objective
Function
Error
traffic light: -1
It's like walking in the mountains in a fog
and following the direction of steepest
descent to reach the village in the valley
But each sample gives us a noisy
estimate of the direction. So our path is
a bit random.
Stochastic Gradient Descent (SGD)
W ← W − η
i
i
∂ L ( W , X )
∂ W i
8
Computing Gradients by Back-Propagation
C(X,Y,Θ)
Cost
W n
dC/dW n
W i
dC /dW i
Fn(Xn-1,Wn)
dC/dXi
Xi
Fi(Xi-1,Wi)
dC /d X i-
Publicité
1
X i-1
F1(X0,W1)
● A practical Application of Chain Rule
● Backprop for the state gradients:
●
●
dC/dXi-1 = dC/dXi . dXi/dXi-1
dC/dXi-1 = dC/dXi . dFi(Xi-1,Wi)/dXi-1
● Backprop for the weight gradients:
●
●
dC/dWi = dC/dXi . dXi/dWi
dC/dWi = dC/dXi . dFi(Xi-1,Wi)/dWi
X (input) Y (desired output)
9
Hubel & Wiesel's Model of the Architecture of the Visual Cortex
[Hubel & Wiesel 1962]:
simple cells detect local features
complex cells “pool” the outputs
of simple cells within a
retinotopic neighborhood.
[Fukushima 1982][LeCun 1989, 1998],[Riesenhuber 1999]......
[Thorpe & Fabre-Thorpe 2001]
“Simple cells”
“Complex
cells”
Multiple
convolutions
pooling
subsampling
10
Convolutional Network Architecture [LeCun et al. NIPS 1989]
Filter Bank +non-linearity
Pooling
Filter Bank +non-linearity
Pooling
Filter Bank +non-linearity
Inspired by [Hubel & Wiesel 1962] &
[Fukushima 1982] (Neocognitron):
simple cells detect local features
complex cells “pool” the outputs of simple
cells within a retinotopic neighborhood.
11
Convolutional Network (LeNet5, vintage 1990)
Filters-tanh → pooling → filters-tanh → pooling → filters-tanh
12
ConvNets can recognize multiple objects
All layers are convolutional
Networks performs simultaneous segmentation and recognition
[LeCun, Bottou, Bengio, Haffner, Proc IEEE 1998]
13
Face & Pedestrian Detection with ConvNets (1993-2005)
[Osadchy,Miller LeCun JMLR 2007],[Kavukcuoglu et al. NIPS 2010] [Sermanet et al. CVPR 2013] 14
Training a Robot to Drive Itself in Nature [Hadsell 2009]
15
Semantic Segmentation with ConvNets [Farabet 2012]
33 categories
16
1986-1996 Neural Net Hardware at Bell Labs, Holmdel
1986: 12x12 resistor array
Fixed resistor values
E-beam lithography: 6x6microns
1988: 54x54 neural net
Programmable ternary weights
On-chip amplifiers and I/O
1991: Net32k: 256x128 net
Programmable ternary weights
320GOPS, 1-bit convolver.
1992: ANNA: 64x64 net
ConvNet accelerator: 4GOPS
6-bit weights, 3-bit activations
6 microns
17
FPGA ConvNet Accelerator: NewFlow [Farabet 2011]
NeuFlow: Reconfigurable Dataflow architecture
Implemented on Xilinx Virtex6 FPGA
20 configurable tiles. 150GOPS, 10 Watts
Semantic Segmentation: 20 frames/sec at 320x240
Exploits the structure of convolutions
NeuFlow ASIC [Pham 2012]
150GOPS, 0.5 Watts (simulated)
18
The Deep Learning
Revolution
Speech recognition: 2 0 1 0
Image recognition: 2 0 1 3
Natural language processing: 2 0 1 5
19
Deep ConvNets for Object Recognition (on GPU)
AlexNet [Krizhevsky et al. NIPS 2012], OverFeat [Sermanet et al. 2013]
1 to 10 billion connections, 10 million to 1 billion parameters, 8 to 20 layers.
20
Error Rate on ImageNet
Depth inflation
(Figure: Anirudh Koul)
Deep ConvNets: depth inflation!
VGG
[Simonyan 2013]
GoogLeNet
Szegedy 2014]
ResNet
[He et al. 2015]
DenseNet
[Huang et al 2017]
22
GOPS vs Accuracy on ImageNet vs #Parameters
[Canziani 2016]
ResNet50 and
ResNet100 are used
Publicité
routinely in
production.
Each of the few
billions photos
uploaded on
Facebook every day
goes through a
handful of ConvNets
within 2 seconds.
23
Multilayer Architectures == Compositional Structure of Data
Natural is data is compositional => it is efficiently representable hierarchically
Low-Level
Mid-Level
Hig h-Level
Feature
Feature
Feature
Trainable
Classifier
Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]
24
Progress in Computer Vision
[He 2017]
Mask-RCNN, RetinaNet, feature pyramid network
Mask-RCNN
[He et al. arXiv:1703.06870]
ConvNet produces an object mask
for each region of interest
RetinaNet/FPN
[Lin et al. ArXiv:1708.02002]
one-pass object detection
26
Mask-RCNN Results on COCO dataset
Individual
objects are
segmented.
Mask R-CNN Results on COCO test set
28
Panoptic Feature Pyramid Network
Segments and recognizes
object instances and regions
[Kirillov arXiv:1901.0244]
Detectron2 (FAIR) [Girshick 2019]
Panoptic instance segmentation, (dense) body pose estimation
Open source: https://github.com/facebookresearch/detectron2
Driving Cars with Convolutional Nets
MobilEye
(2015)
NVIDIA
31
3D ConvNet for Medical Image Analysis (NYU)
Segmentation Femur from MR Images
[Deniz et al. Nature 2018]
3D ConvNet for Medical Image Analysis (NYU)
33
Breast Cancer Detection (NYU)
[Wu et al. ArXiv:1903.08297]
https://github.com/nyukat/breast_cancer_classifier
FastMRI (NYU+FAIR): 4x-8x speed up for MRI data acquisition
MRI images subsampled
(in k-space) by 4x and 8x
[Zbontar et al.
ArXiv:1811.08839]
U-Net architecture
4-fold acceleration
8-fold acceleration
K-space masks
35
ConvNets (and Deep Learning) in Physics
Material Science / Molecular dynamics
Protein structure/function prediction
Prediction of material properties
High energy Physics
Jet filtering / analysis
“Deep learning in color: towards automated
quark/gluon jet discrimination”, P Komiske, E
Metodiev, M Schwartz, arXiv:1612.01551
Cosmology / Astrophysics
Infering constants from observations
Statistical studies of galaxies,
Dark matter through gravitational lensing
36
Applications of ConvNets
Self-driving cars, visual perception
Medical signal and image analysis
Radiology, dermatology, EEG/seizure prediction….
Bioinformatics/genomics
Speech recognition
Language translation
Image restoration/manipulation/style transfer
Robotics, manipulation
Physics
High-energy physics, astrophysics
New applications appear every day
E.g. environmental protection,….
37
Applications of Deep Learning
Medical image analysis
Self-driving cars
Accessibility
Face recognition
Language translation
Virtual assistants*
Content Understanding for:
Filtering
Selection/ranking
Search
Games
Security, anomaly detection
Publicité
Diagnosis, prediction
Science!
[Mnih 2015]
[MobilEye]
[Geras 2017]
[Esteva 2017]
38
ConvNets & The Visual System [Yamins et al. PNAS 2014]
Yamins et al. PNAS
2014]
ConvNets as Models of the Visual System?
[Yamins & Di Carlo 2016]
Why does it work so well?
We can approximate any function with two layers
Why do we need layers?
What is so special convolutional networks?
Why do they work so well on natural signals?
The objective function are highly non-convex.
Why doesn’t SGD get trapped in local minima?
The networks are widely over-parameterized.
Why do they not overfit?
41
The world is compositional
Convolutional networks learn hierarchical representations
Upper-layer representation are at a coarse spatial scale
Renormalization group theory
Multi-scale entanglement renormalization ansatz (MERA)
Low-Level
Mid-Level
High-Level
Trainable
Features
Features
Features
Classifier
Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]42
What current deep learning methods enables
What we can have
Safer cars, autonomous cars
Better medical image analysis
Personalized medicine
Adequate language translation
Useful but stupid chatbots
Information search, retrieval, filtering
Numerous applications in energy,
finance, manufacturing,
environmental protection, commerce,
law, artistic creation, games,…..
What we cannot have (yet)
Machines with common sense
Intelligent personal assistants
“Smart” chatbots”
Household robots
Agile and dexterous robots
Artificial General Intelligence
(AGI)
43
Learning
Repre sentations
What are good representations?
Wh y do networks need to be deep?
Deep Learning = Learning Representations/Features
The traditional model of pattern recognition (since the late 50's)
Fixed/engineered features (or fixed kernel) + trainable classifier
hand-crafted
“Simple” Trainable
Feature Extractor
Classifier
End-to-end learning / Feature learning / Deep learning
Trainable features (or kernel) + trainable classifier
Trainable
Feature Extractor
Trainable
Classifier
45
Ideas for “generic” feature extraction
Basic principle:
expanding the dimension of the representation so that things are more
likely to become linearly separable.
- space tiling
- random projections
- polynomial classifier (feature cross-products)
- radial basis functions
- kernel machines
46
Hierarchical representation
Hierarchy of representations with increasing level of abstraction
Each stage is a kind of trainable feature transform
Image recognition
Pixel → edge → texton → motif → part → object
Text
Character → word → word group → clause → sentence → story
Speech
Sample → spectral band → sound → … → phone → phoneme → word
47
Do we really need deep architectures?
Theoretician's dilemma: “We can approximate any function as close as we
want with shallow architecture. Why would we need deep ones?”
kernel machines (and 2-layer neural nets) are “universal”.
Deep learning machines
Deep machines are more efficient for representing certain classes of
functions, particularly those involved in visual recognition
they can represent more complex functions with less “hardware”
We need an efficient parameterization of the class of functions that are useful
for “AI” tasks (vision, audition, NLP...)
48
Why would deep architectures be more efficient?
[Bengio & LeCun 2 0 0 7 “Scaling Learning Algorithms Towards AI”]
A deep architecture trades space for time (or breadth for depth)
more layers (more sequential computation),
but less hardware (less parallel computation).
Example1: N-bit parity
requires N-1 XOR gates in a tree of depth log(N).
Even easier if we use threshold gates
Publicité
requires an exponential number of gates of we restrict ourselves to 2 layers (DNF
formula with exponential number of minterms).
Example2: circuit for addition of 2 N-bit binary numbers
Requires O(N) gates, and O(N) layers using N one-bit adders with ripple carry
propagation.
Requires lots of gates (some polynomial in N) if we restrict ourselves to two layers (e.g.
Disjunctive Normal Form).
Bad news: almost all boolean functions have a DNF formula with an exponential
number of minterms O(2^N).....
49
Which Models are Deep?
2-layer models are not deep (even if you
train the first layer)
Because there is no feature hierarchy
Neural nets with 1 hidden layer are not
deep
SVMs and Kernel methods are not deep
Layer1: kernels; layer2: linear
The first layer is “trained” in with the
simplest unsupervised method ever
devised: using the samples as templates for
the kernel functions.
Classification trees are not deep
No hierarchy of features. All decisions are
made in the input space
50
What are
Good Features?
What are good representations?
Discovering the Hidden Structure in High-Dimensional Data: The manifold
hypothesis
Learning Representations of Data:
Discovering & disentangling the independent explanatory factors
The Manifold Hypothesis:
Natural data lives in a low-dimensional (non-linear) manifold
Because variables in natural data are mutually dependent
52
Discovering the Hidden Structure in High-Dimensional Data
Example: all face images of a person
1000x1000 pixels = 1,000,000 dimensions
But the face has 3 Cartesian coordinates and 3 Euler angles
And humans have less than about 50 muscles in the face
Hence the manifold of face images for a person has <56 dimensions
The perfect representations of a face image:
Its coordinates on the face manifold
Its coordinates away from the manifold
We do not have good and general methods to learn functions that turns an image into
this kind of representation
Ideal
Feature
Extractor
1 . 2
− 3
0 . 2
− 2 . . .[ ]Face/not face
Pose
Lighting
Expression
53
Disentangling factors of variation
The Ideal Disentangling Feature Extractor
Pixel n
Ideal
Feature
Extractor
Pixel 2
Pixel 1
View
Expression
54
Basic Idea for Invariant Feature Learning
Embed the input non-linearly into a high(er) dimensional space
In the new space, things that were non separable may become separable
Pool regions of the new space together
Bringing together things that are semantically similar. Like pooling.
Non-Linear
Function
Pooling
Or
Aggregation
Input
high-dim
Unstable/non-smooth
features
Stable/invariant
features
55
Non-Linear Expansion → Pooling
Entangled data manifolds
Non-Linear Dim
Expansion,
Disentangling
Pooling.
Aggregation
56
Sparse Non-Linear Expansion → Pooling
Use clustering to break things apart, pool together similar things
Clustering,
Quantization,
Sparse Coding
Pooling.
Aggregation
57
Reference:
•
Yann LeCun’s Deep Learning Course at CDS: DS-GA 1008
FAIR: “Giving people the power to share and
connect requires constant innovation…”