Deep Learning and Applications

d

i=1 wi xi + b

(cid:80)

(cid:73) We have d + 1 values to learn (w and b).

(cid:73) A famous algorithm: Perceptron (Rosenblatt, 1960). Can be seen as the

most simple kind of neural network.

Perceptron algorithm

Input: Training set T , learning rate > 0

begin

k

repeat

maxn

0, R

i=1 (cid:107)

xi(cid:107)2

0, wk

for i

(cid:126)0, bk

1 to n do

if yi (

wk, xi(cid:105)

(cid:104)

wk+1

bk+1

k

k + 1

until no mistake is made

return k, wk , bk

0 then

+ bk )

wk + yi xi

bk + yi R 2

d

Example

= R2.

(cid:73) Let

(cid:73) Assume at step k we have wk = (1 -1)t and bk = 1,

X

( t denotes the vector transpose notation).

(cid:73) Consider the new example (xi , yi ) with xi = (3 2)t and

yi =

1

(cid:73) take = 0.1 and R = 5 what is the new hyperplane?

Example

= R2.

X

(cid:73) Let

(cid:73) Assume at step k we have wk = (1 -1)t and bk = 1

(cid:73) Consider the new example (xi , (cid:96)i ) with xi = (3 2)t and

yi =

1

(cid:73) take = 0.1 and R = 5 what is the new hyperplane?

(cid:73) we have yi (

wk, xi(cid:105)

(cid:104)

error, so the classier must be updated

+ bk ) =

1(1

3

2 + 1) =

1

2 < 0:

Example

= R2.

X

(cid:73) Let

(cid:73) Assume at step k we have wk = (1 -1)t and bk = 1

(cid:73) Consider the new example (xi , yi ) with xi = (3 2)t and

yi =

1

(cid:73) take = 0.1 and R = 5 what is the new hyperplane?

(cid:73) we have yi (

wk, xi(cid:105)

(cid:104)

error, so the classier must be updated

+ bk ) =

1(1

3

2 + 1) =

1

2 < 0:

(cid:73) wk+1 = wk + yi xi = (1 -1)t +0.1

1)

(

(3 2)t = (0.7 -1.2)t

Example

= R2.

X

(cid:73) Let

(cid:73) Assume at step k we have wk = (1 -1)t and bk = 1

(cid:73) Consider the new example (xi , yi ) with xi = (3 2)t and

yi =

1

(cid:73) take = 0.1 and R = 5 what is the new hyperplane?

(cid:73) we have yi (

wk, xi(cid:105)

(cid:104)

error, so the classier must be updated

+ bk ) =

1(1

3

2 + 1) =

1

2 < 0:

(cid:73) wk+1 = wk + yi xi = (1 -1)t +0.1

(cid:73) bk+1 = bk + yi R 2 = 1 + 0.1

(

1)

(

1)

52 =

1.5

(3 2)t = (0.7 -1.2)t

Example

= R2.

X

(cid:73) Let

(cid:73) Assume at step k we have wk = (1 -1)t and bk = 1

(cid:73) Consider the new example (xi , yi ) with xi = (3 2)t and

yi =

1

(cid:73) take = 0.1 and R = 5 what is the new hyperplane?

(cid:73) we have yi (

wk, xi(cid:105)

(cid:104)

error, so the classier must be updated

+ bk ) =

1(1

3

2 + 1) =

1

2 < 0:

(cid:73) wk+1 = wk + yi xi = (1 -1)t +0.1

(cid:73) bk+1 = bk + yi R 2 = 1 + 0.1

(

(cid:73) yi (

3

wk+1, xi(cid:105)

(cid:104)

1

1.8 > 0: correct classication

+ bk+1) =

1(0.7

1)

1)

(

52 =

2

1.2

1.5

1.5) =

(3 2)t = (0.7 -1.2)t

Learning strategy: a bit old school

Today Learning means optimize!

(Next part borrowed from S. Canus presentation and F. Fleuret

slides on SGD)

Costminimization(energy-basedmodel)MinimizealossminwIRp+1nXi=1loss(w)loss(w)=(cid:0) (wtxi)ti(cid:1)2Gradientdescentw w dd=nXi=1wloss(w)Stochasticgradientd=wloss(w)Algorithm1GradientepochData:winitialization, stepsizeResult:wfori=1,ndoxi,ti pickapointid d+wloss(w,xi,ti)endw w dAlgorithm2StochasticgradientData:winitialization, stepsizeResult:wfori=1,ndoxi,ti pickapointid wloss(w,xi,ti)w w dendGradientvs.stochasticgradienttradingprecisionforcomputationaleciencyminimizingthegeneralizationerror(earlystopping)slowertooptimizebutfastertolearnalargescalelearningperspective:O(n)https://wikidocs.net/3413Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections.-3-2-1 0 1 2 3-3-2-1 0 1 2 3-50 0 50 100 150 200 250 300 350Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections. =1.0e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections. =1.0e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections. =2.0e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections. =4.0e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections. =5.0e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Thegradientdescentmethodmakesastrongassumptionaboutthemagnitudeofthelocalcurvaturetoxthestepsize,andaboutitsisotropysothatthesamestepsizemakessenseinalldirections. =5.3e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent9/17Someoptimizationmethodsleveragehigher-ordermoments,inparticularsecondordertouseamoreaccuratelocalmodelofthefunctionaltooptimize.Howeverforaxedcomputationalbudget,thecomplexityofthesemethodsreducesthetotalnumberofiterations,andtheeventualoptimizationisworse.Deep-learninggenerallyreliesonasmarteruseofthegradient,usingstatisticsoveritspastvaluestomakeasmarterstepwiththecurrentone.Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent10/17MomentumandmomentestimationFran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent11/17Thevanillamini-batchstochasticgradientdescent(SGD)consistsofwt+1=wt gt,wheregt=BXb=15 n(t,b)(wt)isthegradientsummedoveramini-batch.Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent12/17Therstimprovementistheuseofamomentumtoaddinertiainthechoiceofthestepdirectionut= ut1+ gtwt+1=wtut.(Rumelhartetal.,1986)With =0,thisisthesameasvanillaSGD.With >0,thisupdatehasthreeniceproperties:"itcangothroughlocalbarriers,"itacceleratesifthegradientdoesnotchangemuch:(u= u+ g) (cid:18)u= 1 g(cid:19),"itdampensoscillationsinnarrowvalleys.Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent13/17 =5.0e2, =0-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent14/17 =5.0e2, =0.5-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent14/17Anotherclassofmethodsexploitsthestatisticsoverthepreviousstepstocompensatefortheanisotropyofthemapping.TheAdamalgorithmusesmovingaveragesofeachcoordinateanditssquaretorescaleeachcoordinateseparately.Theupdateruleis,oneachcoordinateseparatelymt= 1mt1+(1 1)gt mt=mt1 t1vt= 2vt1+(1 2)g2t vt=vt1 t2wt+1=wt vt+(cid:15) mt(KingmaandBa,2014)Thiscanbeseenasacombinationofmomentum,with mt,andaper-coordinatere-scalingwith vt.Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent15/17Anotherclassofmethodsexploitsthestatisticsoverthepreviousstepstocompensatefortheanisotropyofthemapping.TheAdamalgorithmusesmovingaveragesofeachcoordinateanditssquaretorescaleeachcoordinateseparately.Theupdateruleis,oneachcoordinateseparatelymt= 1mt1+(1 1)gt mt=mt1 t1vt= 2vt1+(1 2)g2t vt=vt1 t2wt+1=wt vt+(cid:15) mt(KingmaandBa,2014)Thiscanbeseenasacombinationofmomentum,with mt,andaper-coordinatere-scalingwith vt.Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent15/17 =5.0e2-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent16/17Adam, 1=0.9, 2=0.999,(cid:15)=1e8, =1.0e1-3-2-1 0 1 2 3-3-2-1 0 1 2 3Fran coisFleuretEE-559Deeplearning/5.2.Stochasticgradientdescent16/17Acceleratingthestochasticgradientstochasticaverage(minibatch)Iparameters(PolyakandJuditsky,1992)IgradientsSAG-A,(LeRouxetal2012)Ivariancereduction(Johnson,Zhang,2013)convergenceaccelerationINesterovsmethod(1983)Imomentum(heuristic)accelerationandaveragingI(Dieuleveut,Flammarion&Bach,2016)stepsizeadaptationIRMSprop(Tieleman&Hinton,2012)IAdaptiveMomentEstimationADAM(Kingma&Ba,2015)IAMSGRAD(Reddietal,BPAICRL2018)http://ruder.io/optimizing-gradient-descent/https://wikidocs.net/3413Optimizationmethodsforlarge-scalemachinelearningLBottou,FECurtis,JNocedal-stat,2017-Averagingthroughexamples(minibatch)Forminibatchofsizem(typicallym{32,256})d=m<<nXi=1wloss(w)Algorithm4MinibatchgradientepochData:winitialization, stepsizeResult:wfori=1,ndoxi,ti pickapointid d+wloss(w,xi,ti)ifnmodm==0thenw w dd 0withmomentum:d dendendAcceleratingthestochasticgradientconvergenceacceleration:usevelocity=wtwt1Istochasticgradientwithmomentum(MOM) =.9:d= d+wL(w)w=w dINesterovacceleratedgradient(NAG,1982) =.5:gt=w wL(w)w=gt (gtgt1)stepsizeadaptationIAdaptivestepsizegradient =107 = 0kdk+ IRMSprop(TielemanandHinton,2012) =0.9g= g+(1 )d>d = 0g+ IAdam(KingmaandBa,2015) 1=0.9, 2=0.999h= 1h+(1 )dg= 2g+(1 )d>dw=w 0g+ hFeed-forward Neural Networks

Articial Neuron/Perceptron

We assume to have instances x such that x

Rd

A linear machine (pre-activation or input activation)

d

a(x) =

i=1 wi xi + b

+ b =

w, x

(cid:104)

(cid:105)

(cid:80)

Neuron (output) activation

h(x) = g (a(x)) = g (

i wi xi + b)

(cid:73) w are the connection weights

(cid:80)

(cid:73) b the neuron bias

(cid:73) g (

) the activation function,

sign(.), tanh(.), sigmoid, ...

Articial Neuron - activation

ARTIFICIAL NEURON14Topics: connection weights, bias, activation function2R eseauxdeneurones-11-111111.5-1.5.7-.4-1x1x2x1x2z=+1z=-1z=-101-101-101-101-101-101-101-101-101-1R2R2R1y1y2zzkwkjwjix1x2x1x2x1x2y1y2sortiekentr eeicach eejbiais(from Pascal Vincents slides)FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=biPiwixi=bi+w>x"h(x)=g(a(x))=g(biPiwixi)"w1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=biPiwixi=bi+w>x"h(x)=g(a(x))=g(biPiwixi)"w"{1range determined (by bias only changes the position of the riffFeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"w"{"g( )b1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"w"{"g( )b1REMINDERArticial Neuron - activation

Linear activation function g (a) = a

(cid:73) No restriction on the values

(cid:73) Not very interesting

ACTIVATION FUNCTION9Topics: linear activation function"Performs no input(squashing"Not very interesting...FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"sigm(a)=11+exp(a)"tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1Articial Neuron - activation

sigmoid activation function - g (a) = sigmoid(a) =

1

1+exp(

a)

(cid:73) Restrict neurons activation in the interval [0, 1]

(cid:73) Always positive

(cid:73) Bounded

(cid:73) Strictly increasing

ACTIVATION FUNCTION10Topics: sigmoid activation function"Squashes the neurons(pre-activation between (0 and 1"Always positive"Bounded"Strictly increasingFeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1Articial Neuron - activation

hyperbolic tangent activation function - g (a) = tanh(a)

(cid:73) Restrict neurons activation in the interval [

1, 1]

(cid:73) Bounded and Strictly increasing, can be positive or negative

exp(

tanh(a) = exp(a)

exp(a)+exp(

a)

a) = exp(2a)

1

exp(2a)+1

ACTIVATION FUNCTION11Topics: hyperbolic tangent (tanh) activation function"Squashes the neurons(pre-activation between (-1 and 1"Can be positive or(negative "Bounded"Strictly increasingFeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1Articial Neuron - activation

Rectied linear (ReLu) activation function -

g (a) = reclin(a) = max(1, 0)

(cid:73) Restrict neurons activation in the interval [

1, 1]

(cid:73) Always non-negative (bounded below by 0)

(cid:73) Not upperbounded

(cid:73) Tends to give neurons with sparse activities

ACTIVATION FUNCTION12Topics: rectied linear activation function"Bounded below by 0((always non-negative)"Not upper bounded"Strictly increasing"Tends to give neurons(with sparse activities FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1Neuron Activation

from Hugo Larochelle and Pascal Vincent

ARTIFICIAL NEURON14Topics: connection weights, bias, activation function2R eseauxdeneurones-11-111111.5-1.5.7-.4-1x1x2x1x2z=+1z=-1z=-101-101-101-101-101-101-101-101-101-1R2R2R1y1y2zzkwkjwjix1x2x1x2x1x2y1y2sortiekentr eeicach eejbiais(from Pascal Vincents slides)FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=biPiwixi=bi+w>x"h(x)=g(a(x))=g(biPiwixi)"w1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=biPiwixi=bi+w>x"h(x)=g(a(x))=g(biPiwixi)"w"{1range determined (by bias only changes the position of the riffFeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"w"{"g( )b1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"w"{"g( )b1REMINDERCapacity, decision frontier of one neuron

Can do binary classication

The sigmoid activation function can interpret neuron as estimating

x) (similar idea can apply with tanh)

p(y = 1

|

(cid:73) Also known as Logistic regression classier

if greater than 0.5 predict class 1

Otherwise, predict class 0.

decision boundary is linear

(cid:73)

(cid:73)

from Hugo Larochelle and Pascal Vincent

ARTIFICIAL NEURON15Topics: capacity, decision boundary of neuron"Could do binary classication:#with sigmoid, can interpret neuron as estimating #also known as logistic regression classier#if greater than 0.5,(predict class 1#otherwise, predict(class 0(((((similar idea can( apply with tanh)(from Pascal Vincents slides)7R eseauxdeneurones"Lapuissanceexpressivedesr eseauxdeneuronesx1x2x1x2...x1x2R1R2R1R2R2R1x2x1deuxcouchestroiscouchesdecision boundary is linearFeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"p(y=1|x)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)i+PjW(1)i,jxj"f(x)=o b(2)+w(2)>x"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>1Capacity of a single neuron

Can solve linearly separable problems

from Hugo Larochelle and Pascal Vincent

ARTIFICIAL NEURON16Topics: capacity of single neuron"Can solve linearly separable problems210101010101010101?XOR(x1,x2)OR(x1,x2)AND(x1,x2)AND(x1,x2)(x1(x1,x2)0101XOR(x1,x2)AND(x1,x2)AND(x1,x2),x2),x2)(x1(x1,x2)Figure1.8Exempledemod lisationdeXORparunr seau unecouchecach e.Enhaut,degauche droite,illustrationdesfonctionsbool ennesOR(x1,x2),AND(x1,x2)etAND(x1,x2).Enbas,onpr sentelillustrationdelafonctionXOR(x1,x2)enfonc-tiondesvaleursdex1etx2( gauche),puisdeAND(x1,x2)etAND(x1,x2)( droite).Lespointsrepr sent sparuncercleouparuntriangleappartiennent laclasse0ou1,respectivement.Onobserveque,bienquunclassieurlin airesoitenmesureder -soudreleprobl medeclassicationassoci auxfonctionsORetAND,ilnelestpasdanslecasduprobl medeXOR.Cependant,onutilisantlesvaleursdeAND(x1,x2)etAND(x1,x2)commenouvellerepr sentationdelentr e(x1,x2),leprobl medeclassicationXORpeutalors trer solulin airement. noterquedanscederniercas,ilnexistequetroisvaleurspossiblesdecettenouvellerepr sentation,puisqueAND(x1,x2)etAND(x1,x2)nepeuvent tretouteslesdeuxvraiespourunem meentr e.ont t entra n sdanslecadredestravauxdecetteth secontiennentquelquesmilliersdeneuronescach s.Ainsi,lat chedelalgorithmedapprentissageestdemodierlesparam tresdur seauandetrouverlanaturedescaract ristiquesdelentr equechaqueneuronedoitextrairepourr soudreleprobl medeclassication.Id alement,cescarac-Capacity of a single neuron

But cant solve non linearly separable problems

Winter

1st NN

unless the input is transformed in a better representation.

from Hugo Larochelle and Pascal Vincent

ARTIFICIAL NEURON17Topics: capacity of single neuron"Cant solve non linearly separable problems..."... unless the input is transformed in a better representation210101010101010101?XOR(x1,x2)OR(x1,x2)AND(x1,x2)AND(x1,x2)(x1(x1,x2)0101XOR(x1,x2)AND(x1,x2)AND(x1,x2),x2),x2)(x1(x1,x2)Figure1.8Exempledemod lisationdeXORparunr seau unecouchecach e.Enhaut,degauche droite,illustrationdesfonctionsbool ennesOR(x1,x2),AND(x1,x2)etAND(x1,x2).Enbas,onpr sentelillustrationdelafonctionXOR(x1,x2)enfonc-tiondesvaleursdex1etx2( gauche),puisdeAND(x1,x2)etAND(x1,x2)( droite).Lespointsrepr sent sparuncercleouparuntriangleappartiennent laclasse0ou1,respectivement.Onobserveque,bienquunclassieurlin airesoitenmesureder -soudreleprobl medeclassicationassoci auxfonctionsORetAND,ilnelestpasdanslecasduprobl medeXOR.Cependant,onutilisantlesvaleursdeAND(x1,x2)etAND(x1,x2)commenouvellerepr sentationdelentr e(x1,x2),leprobl medeclassicationXORpeutalors trer solulin airement. noterquedanscederniercas,ilnexistequetroisvaleurspossiblesdecettenouvellerepr sentation,puisqueAND(x1,x2)etAND(x1,x2)nepeuvent tretouteslesdeuxvraiespourunem meentr e.ont t entra n sdanslecadredestravauxdecetteth secontiennentquelquesmilliersdeneuronescach s.Ainsi,lat chedelalgorithmedapprentissageestdemodierlesparam tresdur seauandetrouverlanaturedescaract ristiquesdelentr equechaqueneuronedoitextrairepourr soudreleprobl medeclassication.Id alement,cescarac-More expressiveness: add a hidden layer

single hidden layer neural

network

(cid:73) Hidden layer pre-activation:

a(x) = b(1) + W(1)x

a(x)i = b(1)

j W (1)

i,j xj

(cid:73) Hidden layer activation:

i +

(cid:16)

h(x) = g(a(x))

(cid:80)

(cid:17)

(cid:73) Output layer activation:

b(2) + w(2)T

f (x) = o

o is the output activation

function

(cid:16)

h(1)x

(cid:17)

NEURAL NETWORK19Topics: single hidden layer neural network"Hidden layer pre-activation:"Hidden layer activation:"Output layer activation:...1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xd"w"{"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1......FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xd"w"{"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1...FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)i"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)i"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)i"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)i"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)11FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"f(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)i+PjW(1)i,jxj"f(x)=o(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)i+PjW(1)i,jxj"f(x)=o(b(2)+w(2)>x)1output activation functionFeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"p(y=1|x)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)i+PjW(1)i,jxj"f(x)=o b(2)+w(2)>x1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"p(y=1|x)"g( )b"W(1)i,jb(1)ixjh(x)iw(2)ib(2)"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)i+PjW(1)i,jxj"f(x)=o b(2)+w(2)>h(1)x1Softmax activation function

For multi-class classication

(cid:73) Need of multiple outputs (1 per class)

(cid:73) We would like to estimate the conditional probability

p(y = c

x)

|

Softmax activation function at the output - for c classes

o(a) = softmax(a) =

strictly positive and sums up to one

(cid:104)

(cid:80)

exp(a1)

c exp(ac ) ,

,

exp(aC )

c exp(ac )

(cid:80)

(cid:105)

Predicted class is the one with highest estimated probability.

More expressiveness: MultiLayer neural network (MLP)

L hidden layers

(cid:73) layer pre-activation for k > 0

(h(0)(x) = x)

a(k)(x) = W(k)h(k

1)(x) + b(k)

(cid:73) hidden layer activation (k from

1 to L):

h(k)(x) = g(a(k)(x))

(cid:73) output layer activation

(k=L+1)

hL+1(x) = o(aL+1(x)) = f(x)

NEURAL NETWORK21Topics: multilayer neural network"Could have L hidden layers:#layer pre-activation for k>0#hidden layer activation (k from 1 to L):#output layer activation (k=L+1):...FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xd"w"{"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xd"w"{"g( )b"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)1...FeedforwardneuralnetworkHugoLarochelleD epartementdinformatiqueUniversit [email protected],2012AbstractMathformyslidesFeedforwardneuralnetwork."a(x)=b+Piwixi=b+w>x"h(x)=g(a(x))=g(b+Piwixi)"x1xdbw1wd"w"{"g(a)=a"g(a)=sigm(a)=11+exp(a)"g(a)=tanh(a)=exp(a)exp(a)exp(a)+exp(a)=exp(2a)1exp(2a)+1"g(a)=max(0,a)"g(a)=reclin(a)=max(0,a)"g( )b"W(1)i,jb(1)ixjh(x)i"h(x)=g(a(x))"a(x)=b(1)+W(1)x a(x)i=b(1)iPjW(1)i,jxj"o(x)=g(out)(b(2)+w(2)>x)111......1........."p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)x(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2"p(y=c|x)"o(a)=softmax(a)=hexp(a1)Pcexp(ac)...exp(aC)Pcexp(ac)i>"f(x)"h(1)(x)h(2)(x)W(1)W(2)W(3)b(1)b(2)b(3)"a(k)(x)=b(k)+W(k)h(k1)(x)(h(0)(x)=x)"h(k)(x)=g(a(k)(x))"h(L+1)(x)=o(a(L+1)(x))=f(x)2A full computation with 2-Layers

2 hidden layers

(cid:73) layer pre-activation for k > 0

(h(0)(x) = x)

a(k)(x) = W(k)h(k

1)(x) + b(k)

(cid:73) hidden layer activation (k from 1 to

L):

h(k)(x) = g(a(k)(x))

(cid:73) output layer activation (k=L+1)

<...