Deep Learning

1

X i-1

F1(X0,W1)

● A practical Application of Chain Rule

● Backprop for the state gradients:

●

●

dC/dXi-1 = dC/dXi . dXi/dXi-1

dC/dXi-1 = dC/dXi . dFi(Xi-1,Wi)/dXi-1

● Backprop for the weight gradients:

●

●

dC/dWi = dC/dXi . dXi/dWi

dC/dWi = dC/dXi . dFi(Xi-1,Wi)/dWi

X (input) Y (desired output)

9

Hubel & Wiesel's Model of the Architecture of the Visual Cortex

[Hubel & Wiesel 1962]:

simple cells detect local features

complex cells “pool” the outputs

of simple cells within a

retinotopic neighborhood.

[Fukushima 1982][LeCun 1989, 1998],[Riesenhuber 1999]......

[Thorpe & Fabre-Thorpe 2001]

“Simple cells”

“Complex

cells”

Multiple

convolutions

pooling

subsampling

10

Convolutional Network Architecture [LeCun et al. NIPS 1989]

Filter Bank +non-linearity

Pooling

Filter Bank +non-linearity

Pooling

Filter Bank +non-linearity

Inspired by [Hubel & Wiesel 1962] &

[Fukushima 1982] (Neocognitron):

simple cells detect local features

complex cells “pool” the outputs of simple

cells within a retinotopic neighborhood.

11

Convolutional Network (LeNet5, vintage 1990)

Filters-tanh → pooling → filters-tanh → pooling → filters-tanh

12

ConvNets can recognize multiple objects

All layers are convolutional

Networks performs simultaneous segmentation and recognition

[LeCun, Bottou, Bengio, Haffner, Proc IEEE 1998]

13

Face & Pedestrian Detection with ConvNets (1993-2005)

[Osadchy,Miller LeCun JMLR 2007],[Kavukcuoglu et al. NIPS 2010] [Sermanet et al. CVPR 2013] 14

Training a Robot to Drive Itself in Nature [Hadsell 2009]

15

Semantic Segmentation with ConvNets [Farabet 2012]

33 categories

16

1986-1996 Neural Net Hardware at Bell Labs, Holmdel

1986: 12x12 resistor array

Fixed resistor values

E-beam lithography: 6x6microns

1988: 54x54 neural net

Programmable ternary weights

On-chip amplifiers and I/O

1991: Net32k: 256x128 net

Programmable ternary weights

320GOPS, 1-bit convolver.

1992: ANNA: 64x64 net

ConvNet accelerator: 4GOPS

6-bit weights, 3-bit activations

6 microns

17

FPGA ConvNet Accelerator: NewFlow [Farabet 2011]

NeuFlow: Reconfigurable Dataflow architecture

Implemented on Xilinx Virtex6 FPGA

20 configurable tiles. 150GOPS, 10 Watts

Semantic Segmentation: 20 frames/sec at 320x240

Exploits the structure of convolutions

NeuFlow ASIC [Pham 2012]

150GOPS, 0.5 Watts (simulated)

18

The Deep Learning

Revolution

Speech recognition: 2 0 1 0

Image recognition: 2 0 1 3

Natural language processing: 2 0 1 5

19

Deep ConvNets for Object Recognition (on GPU)

AlexNet [Krizhevsky et al. NIPS 2012], OverFeat [Sermanet et al. 2013]

1 to 10 billion connections, 10 million to 1 billion parameters, 8 to 20 layers.

20

Error Rate on ImageNet

Depth inflation

(Figure: Anirudh Koul)

Deep ConvNets: depth inflation!

VGG

[Simonyan 2013]

GoogLeNet

Szegedy 2014]

ResNet

[He et al. 2015]

DenseNet

[Huang et al 2017]

22

GOPS vs Accuracy on ImageNet vs #Parameters

[Canziani 2016]

ResNet50 and

ResNet100 are used

routinely in

production.

Each of the few

billions photos

uploaded on

Facebook every day

goes through a

handful of ConvNets

within 2 seconds.

23

Multilayer Architectures == Compositional Structure of Data

Natural is data is compositional => it is efficiently representable hierarchically

Low-Level

Mid-Level

Hig h-Level

Feature

Feature

Feature

Trainable

Classifier

Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]

24

Progress in Computer Vision

[He 2017]

Mask-RCNN, RetinaNet, feature pyramid network

Mask-RCNN

[He et al. arXiv:1703.06870]

ConvNet produces an object mask

for each region of interest

RetinaNet/FPN

[Lin et al. ArXiv:1708.02002]

one-pass object detection

26

Mask-RCNN Results on COCO dataset

Individual

objects are

segmented.

Mask R-CNN Results on COCO test set

28

Panoptic Feature Pyramid Network

Segments and recognizes

object instances and regions

[Kirillov arXiv:1901.0244]

Detectron2 (FAIR) [Girshick 2019]

Panoptic instance segmentation, (dense) body pose estimation

Open source: https://github.com/facebookresearch/detectron2

Driving Cars with Convolutional Nets

MobilEye

(2015)

NVIDIA

31

3D ConvNet for Medical Image Analysis (NYU)

Segmentation Femur from MR Images

[Deniz et al. Nature 2018]

3D ConvNet for Medical Image Analysis (NYU)

33

Breast Cancer Detection (NYU)

[Wu et al. ArXiv:1903.08297]

https://github.com/nyukat/breast_cancer_classifier

FastMRI (NYU+FAIR): 4x-8x speed up for MRI data acquisition

MRI images subsampled

(in k-space) by 4x and 8x

[Zbontar et al.

ArXiv:1811.08839]

U-Net architecture

4-fold acceleration

8-fold acceleration

K-space masks

35

ConvNets (and Deep Learning) in Physics

Material Science / Molecular dynamics

Protein structure/function prediction

Prediction of material properties

High energy Physics

Jet filtering / analysis

“Deep learning in color: towards automated

quark/gluon jet discrimination”, P Komiske, E

Metodiev, M Schwartz, arXiv:1612.01551

Cosmology / Astrophysics

Infering constants from observations

Statistical studies of galaxies,

Dark matter through gravitational lensing

36

Applications of ConvNets

Self-driving cars, visual perception

Medical signal and image analysis

Radiology, dermatology, EEG/seizure prediction….

Bioinformatics/genomics

Speech recognition

Language translation

Image restoration/manipulation/style transfer

Robotics, manipulation

Physics

High-energy physics, astrophysics

New applications appear every day

E.g. environmental protection,….

37

Applications of Deep Learning

Medical image analysis

Self-driving cars

Accessibility

Face recognition

Language translation

Virtual assistants*

Content Understanding for:

Filtering

Selection/ranking

Search

Games

Security, anomaly detection

Diagnosis, prediction

Science!

[Mnih 2015]

[MobilEye]

[Geras 2017]

[Esteva 2017]

38

ConvNets & The Visual System [Yamins et al. PNAS 2014]

Yamins et al. PNAS

2014]

ConvNets as Models of the Visual System?

[Yamins & Di Carlo 2016]

Why does it work so well?

We can approximate any function with two layers

Why do we need layers?

What is so special convolutional networks?

Why do they work so well on natural signals?

The objective function are highly non-convex.

Why doesn’t SGD get trapped in local minima?

The networks are widely over-parameterized.

Why do they not overfit?

41

The world is compositional

Convolutional networks learn hierarchical representations

Upper-layer representation are at a coarse spatial scale

Renormalization group theory

Multi-scale entanglement renormalization ansatz (MERA)

Low-Level

Mid-Level

High-Level

Trainable

Features

Features

Features

Classifier

Feature visualization of convolutional net trained on ImageNet from [Zeiler & Fergus 2013]42

What current deep learning methods enables

What we can have

Safer cars, autonomous cars

Better medical image analysis

Personalized medicine

Adequate language translation

Useful but stupid chatbots

Information search, retrieval, filtering

Numerous applications in energy,

finance, manufacturing,

environmental protection, commerce,

law, artistic creation, games,…..

What we cannot have (yet)

Machines with common sense

Intelligent personal assistants

“Smart” chatbots”

Household robots

Agile and dexterous robots

Artificial General Intelligence

(AGI)

43

Learning

Repre sentations

What are good representations?

Wh y do networks need to be deep?

Deep Learning = Learning Representations/Features

The traditional model of pattern recognition (since the late 50's)

Fixed/engineered features (or fixed kernel) + trainable classifier

hand-crafted

“Simple” Trainable

Feature Extractor

Classifier

End-to-end learning / Feature learning / Deep learning

Trainable features (or kernel) + trainable classifier

Trainable

Feature Extractor

Trainable

Classifier

45

Ideas for “generic” feature extraction

Basic principle:

expanding the dimension of the representation so that things are more

likely to become linearly separable.

• space tiling
• random projections
• polynomial classifier (feature cross-products)
• radial basis functions
• kernel machines

46

Hierarchical representation

Hierarchy of representations with increasing level of abstraction

Each stage is a kind of trainable feature transform

Image recognition

Pixel → edge → texton → motif → part → object

Text

Character → word → word group → clause → sentence → story

Speech

Sample → spectral band → sound → … → phone → phoneme → word

47

Do we really need deep architectures?

Theoretician's dilemma: “We can approximate any function as close as we

want with shallow architecture. Why would we need deep ones?”

kernel machines (and 2-layer neural nets) are “universal”.

Deep learning machines

Deep machines are more efficient for representing certain classes of

functions, particularly those involved in visual recognition

they can represent more complex functions with less “hardware”

We need an efficient parameterization of the class of functions that are useful

for “AI” tasks (vision, audition, NLP...)

48

Why would deep architectures be more efficient?

[Bengio & LeCun 2 0 0 7 “Scaling Learning Algorithms Towards AI”]

A deep architecture trades space for time (or breadth for depth)

more layers (more sequential computation),

but less hardware (less parallel computation).

Example1: N-bit parity

requires N-1 XOR gates in a tree of depth log(N).

Even easier if we use threshold gates

requires an exponential number of gates of we restrict ourselves to 2 layers (DNF

formula with exponential number of minterms).

Example2: circuit for addition of 2 N-bit binary numbers

Requires O(N) gates, and O(N) layers using N one-bit adders with ripple carry

propagation.

Requires lots of gates (some polynomial in N) if we restrict ourselves to two layers (e.g.

Disjunctive Normal Form).

Bad news: almost all boolean functions have a DNF formula with an exponential

number of minterms O(2^N).....

49

Which Models are Deep?

2-layer models are not deep (even if you

train the first layer)

Because there is no feature hierarchy

Neural nets with 1 hidden layer are not

deep

SVMs and Kernel methods are not deep

Layer1: kernels; layer2: linear

The first layer is “trained” in with the

simplest unsupervised method ever

devised: using the samples as templates for

the kernel functions.

Classification trees are not deep

No hierarchy of features. All decisions are

made in the input space

50

What are

Good Features?

What are good representations?

Discovering the Hidden Structure in High-Dimensional Data: The manifold

hypothesis

Learning Representations of Data:

Discovering & disentangling the independent explanatory factors

The Manifold Hypothesis:

Natural data lives in a low-dimensional (non-linear) manifold

Because variables in natural data are mutually dependent

52

Discovering the Hidden Structure in High-Dimensional Data

Example: all face images of a person

1000x1000 pixels = 1,000,000 dimensions

But the face has 3 Cartesian coordinates and 3 Euler angles

And humans have less than about 50 muscles in the face

Hence the manifold of face images for a person has <56 dimensions

The perfect representations of a face image:

Its coordinates on the face manifold

Its coordinates away from the manifold

We do not have good and general methods to learn functions that turns an image into

this kind of representation

Ideal

Feature

Extractor

1 . 2

− 3

0 . 2

− 2 . . .[ ]Face/not face

Pose

Lighting

Expression

53

Disentangling factors of variation

The Ideal Disentangling Feature Extractor

Pixel n

Ideal

Feature

Extractor

Pixel 2

Pixel 1

View

Expression

54

Basic Idea for Invariant Feature Learning

Embed the input non-linearly into a high(er) dimensional space

In the new space, things that were non separable may become separable

Pool regions of the new space together

Bringing together things that are semantically similar. Like pooling.

Non-Linear

Function

Pooling

Or

Aggregation

Input

high-dim

Unstable/non-smooth

features

Stable/invariant

features

55

Non-Linear Expansion → Pooling

Entangled data manifolds

Non-Linear Dim

Expansion,

Disentangling

Pooling.

Aggregation

56

Sparse Non-Linear Expansion → Pooling

Use clustering to break things apart, pool together similar things

Clustering,

Quantization,

Sparse Coding

Pooling.

Aggregation

57

Reference:

•

Yann LeCun’s Deep Learning Course at CDS: DS-GA 1008

FAIR: “Giving people the power to share and

connect requires constant innovation…”