Deep Learning and Applications

∼

X

(cid:107)

g

2

f (x)

(cid:107)

◦

−

0

(cid:39)

If f and g are parametrized mappings by wf and wg respectively,

then training consists of minimizing an empirical estimate of that

loss - unsupervised training

(cid:3)

(cid:2)

ˆwf , ˆwg = argminwf ,wg

1

N

N

(cid:88)i=1

xi −

(cid:107)

2

gwg (fwf (xi ))

(cid:107)

With both f and g linear, an optimal solution can be given by

PCA!

Deep Autoencoder

More sophisticated classes of mappings can be obtained with Deep

architectures.

A linear auto-encoder where the

weights are tied between encoder and

decoder

More layers below:

AUTOENCODER96Topics: autoencoder, encoder, decoder, tied weights•Feed-forward neural network trained to reproduce its input at the output layerDecoderEncoderbjckxxWWAutoencodersHugoLarochelleD´epartementd’informatiqueUniversit´[email protected],2012AbstractMathformyslides“Autoencoders”.•h(x)=g(a(x))=sigm(b+Wx)•bx=o(ba(x))=sigm(c+W⇤h(x))•f(x)⌘bxl(f(x))=Pk(bxkxk)2l(f(x))=Pk(xklog(bxk)+(1xk)log(1bxk))1AutoencodersHugoLarochelleD´epartementd’informatiqueUniversit´[email protected],2012AbstractMathformyslides“Autoencoders”.•h(x)=g(a(x))=sigm(b+Wx)•bx=o(ba(x))=sigm(c+W⇤h(x))•f(x)⌘bxl(f(x))=Pk(bxkxk)2l(f(x))=Pk(xklog(bxk)+(1xk)log(1bxk))1for binary inputs=W(tied weights)AutoencodersHugoLarochelleD´epartementd’informatiqueUniversit´[email protected],2012AbstractMathformyslides“Autoencoders”.•h(x)=g(a(x))=sigm(b+Wx)•bx=o(ba(x))=sigm(c+W⇤h(x))•f(x)⌘bxl(f(x))=12Pk(bxkxk)2l(f(x))=Pk(xklog(bxk)+(1xk)log(1bxk))•rba(x(t))l(f(x(t)))=bx(t)x(t)a(x(t))(=b+Wx(t)h(x(t))(=sigm(a(x(t)))ba(x(t))(=c+W>h(x(t))bx(t)(=sigm(ba(x(t)))rba(x(t))l(f(x(t)))(=bx(t)x(t)rcl(f(x(t)))(=rba(x(t))l(f(x(t)))rh(x(t))l(f(x(t)))(=W⇣rba(x(t))l(f(x(t)))⌘ra(x(t))l(f(x(t)))(=⇣rh(x(t))l(f(x(t)))⌘[...,h(x(t))j(1h(x(t))j),...]rbl(f(x(t)))(=ra(x(t))l(f(x(t)))rWl(f(x(t)))(=⇣ra(x(t))l(f(x(t)))⌘x(t)>+h(x(t))⇣rba(x(t))l(f(x(t)))⌘>•W⇤=W>1Autoencoder-Fully-connected AutoencoderConvolutional Auto-encoder

The decoder ”mirrors” the encoder, we use deconvolution and

unpooling/upsampling layers.

New layers:-Deconvolutional layers-Unpooling (upsampling) layersAutoencoderDenoising auto-encoders

-Strictly speaking, denoising autoencoder is not an autoencoder-It performs some useful transformation on the inputDenoising AutoencoderDenoising AE

Idea

(cid:73) Representation should be robust to noise, e.g.:

(cid:73) Move to zero some input according to a distribution proba p

(cid:73) Additive Gaussian noise: xi →

xi + (cid:15)i where (cid:15)i ∼ N

(µ, Σ)

(cid:73) Reconstruction from a corrupted input: we learn how to

remove noise

(cid:73) the loss function compares the reconstructed output to the

noiseless input

ˆθ = argminθ

1

N

xi −

(cid:107)

φθ(xi + (cid:15)i )

(cid:107)

2,

N

(cid:88)i=1

where φ represents g

◦

f and θ the associated parameters

Illustration

DENOISING AUTOENCODER4ExtractingandComposingRobustFeatureswithDenoisingAutoencodersp(eX|X)=qD(eX|X).p(Y)isauniformprioroverY2[0,1]d0.Thisdefinesagenerativemodelwithpa-rameterset✓0={W0,b0}.Wewillusetheprevi-ouslydefinedq0(X,eX,Y)=q0(X)qD(eX|X)f✓(eX)(Y)(equation4)asanauxiliarymodelinthecontextofavariationalapproximationofthelog-likelihoodofp(eX).Notethatweabusenotationtomakeitlighter,andusethesamelettersX,eXandYfordi↵erentsetsofrandomvariablesrepresentingthesamequan-tityunderdi↵erentdistributions:porq0.KeepinmindthatwhereaswehadthedependencystructureX!eX!Yforqorq0,wehaveY!X!eXforp.Sincepcontainsacorruptionoperationatthelastgenerativestage,weproposetofitp(eX)tocorruptedtrainingsamples.Performingmaximumlikelihoodfit-tingforsamplesdrawnfromq0(eX)correspondstomin-imizingthecross-entropy,ormaximizingH=max✓0{IH(q0(eX)kp(eX))}=max✓0{EEq0(eX)[logp(eX)]}.(6)Letq?(X,Y|eX)beaconditionaldensity,thequan-tityL(q?,eX)=EEq?(X,Y|eX)hlogp(X,eX,Y)q?(X,Y|eX)iisalowerboundonlogp(eX)sincethefollowingcanbeshowntobetrueforanyq?:logp(eX)=L(q?,eX)+IDKL(q?(X,Y|eX)kp(X,Y|eX))Alsoitiseasytoverifythattheboundistightwhenq?(X,Y|eX)=p(X,Y|eX),wheretheIDKLbecomes0.Wecanthuswritelogp(eX)=maxq?L(q?,eX),andconsequentlyrewriteequation6asH=max✓0{EEq0(eX)[maxq?L(q?,eX)]}=max✓0,q?{EEq0(eX)[L(q?,eX)]}(7)xx˜x˜xqD(˜x|x)g✓0(f✓(˜x))Figure2.Manifoldlearningperspective.Supposetrainingdata(⇥)concentratenearalow-dimensionalman-ifold.Corruptedexamples(.)obtainedbyapplyingcor-ruptionprocessqD(eX|X)willliefartherfromthemanifold.Themodellearnswithp(X|eX)to“projectthemback”ontothemanifold.IntermediaterepresentationYcanbeinter-pretedasacoordinatesystemforpointsonthemanifold.wherewemovedthemaximizationoutsideoftheex-pectationbecauseanunconstrainedq?(X,Y|eX)caninprincipleperfectlymodeltheconditionaldistribu-tionneededtomaximizeL(q?,eX)foranyeX.Nowifwereplacethemaximizationoveranunconstrainedq?bythemaximizationovertheparameters✓ofourq0(appearinginf✓thatmapsanxtoay),wegetalowerboundonH:Hmax✓0,✓{EEq0(eX)[L(q0,eX)]}Maximizingthislowerbound,wefindargmax✓,✓0{EEq0(eX)[L(q0,eX)]}=argmax✓,✓0EEq0(X,eX,Y)"logp(X,eX,Y)q0(X,Y|eX)#=argmax✓,✓0EEq0(X,eX,Y)hlogp(X,eX,Y)i+EEq0(eX)hIH[q0(X,Y|eX)]i=argmax✓,✓0EEq0(X,eX,Y)hlogp(X,eX,Y)i.Notethat✓onlyoccursinY=f✓(eX),and✓0onlyoccursinp(X|Y).Thelastlineisthereforeobtainedbecauseq0(X|eX)/qD(eX|X)q0(X)(noneofwhichde-pendson(✓,✓0)),andq0(Y|eX)isdeterministic,i.e.,itsentropyisconstant,irrespectiveof(✓,✓0).Hencetheentropyofq0(X,Y|eX)=q0(Y|eX)q0(X|eX),doesnotvarywith(✓,✓0).Finally,followingfromabove,weobtainourtrainingcriterion(eq.5):argmax✓,✓0EEq0(eX)[L(q0,eX)]=argmax✓,✓0EEq0(X,eX,Y)[log[p(Y)p(X|Y)p(eX|X)]]=argmax✓,✓0EEq0(X,eX,Y)[logp(X|Y)]=argmax✓,✓0EEq0(X,eX)[logp(X|Y=f✓(eX))]=argmin✓,✓0EEq0(X,eX)hLIH⇣X,g✓0(f✓(eX))⌘iwherethethirdlineisobtainedbecause(✓,✓0)havenoinfluenceonEEq0(X,eX,Y)[logp(Y)]becausewechosep(Y)uniform,i.e.constant,noronEEq0(X,eX)[logp(eX|X)],andthelastlineisobtainedbyinspectionofthedefinitionofLIHineq.2,whenp(X|Y=f✓(eX))isaBg✓0(f✓(eX)).4.3.OtherTheoreticalPerspectivesInformationTheoreticPerspective:ConsiderX⇠q(X),qunknown,Y=f✓(eX).Itcaneasilybeshown(Vincentetal.,2008)thatminimizingtheexpectedreconstructionerroramountstomaximizingp(x|x)•rWl(f(x(t)))W•p(x|µ)µ•µh(x)•l(f(x))=logp(x|µ)•p(x|µ)=1(2⇡)D/2exp(12Pk(xkµk)2)µ=c+W⇤h(x)•AA=U⌃V>U·,k⌃k,kV>·,kB•B⇤=argminBs.t.rank(B)=k||AB||FB⇤=U·,k⌃k,kV>·,kargmin✓Xt12Xk(x(t)kbx(t)k)2argminW⇤,h(X)||XW⇤h(X)||F•x(t)X=U⌃V>argminW⇤,h(X)||XW⇤h(X)||F=W⇤ U·,k⌃k,k,h(X) V>·,kh(X)=V>·,k=V>·,k(X>X)1(X>X)=V>·,k(V⌃>U>U⌃V>)1(V⌃>U>X)=V>·,k(V⌃>⌃V>)1V⌃>U>X=V>·,kV(⌃>⌃)1V>V⌃>U>X=V>·,kV(⌃>⌃)1⌃>U>X=Ik,·(⌃>⌃)1⌃>U>X=Ik,·⌃1(⌃>)1⌃>U>X=Ik,·⌃1U>X=⌃1k,k(U·,k)>X•bx=(U·,k⌃k,k)h(x)h(x)=⇣⌃1k,k(U·,k)>⌘xW⇤W•x(t) 1pT⇣x(t)1TPTt0=1x(t0)⌘•p(ex|x)⌫ex•bx=sigm(c+W⇤h(ex))2•rWl(f(x(t)))W•p(x|µ)µ•µh(x)•l(f(x))=logp(x|µ)•p(x|µ)=1(2⇡)D/2exp(12Pk(xkµk)2)µ=c+W⇤h(x)•AA=U⌃V>U·,k⌃k,kV>·,kB•B⇤=argminBs.t.rank(B)=k||AB||FB⇤=U·,k⌃k,kV>·,kargmin✓Xt12Xk(x(t)kbx(t)k)2argminW⇤,h(X)||XW⇤h(X)||F•x(t)X=U⌃V>argminW⇤,h(X)||XW⇤h(X)||F=W⇤ U·,k⌃k,k,h(X) V>·,kh(X)=V>·,k=V>·,k(X>X)1(X>X)=V>·,k(V⌃>U>U⌃V>)1(V⌃>U>X)=V>·,k(V⌃>⌃V>)1V⌃>U>X=V>·,kV(⌃>⌃)1V>V⌃>U>X=V>·,kV(⌃>⌃)1⌃>U>X=Ik,·(⌃>⌃)1⌃>U>X=Ik,·⌃1(⌃>)1⌃>U>X=Ik,·⌃1U>X=⌃1k,k(U·,k)>X•bx=(U·,k⌃k,k)h(x)h(x)=⇣⌃1k,k(U·,k)>⌘xW⇤W•x(t) 1pT⇣x(t)1TPTt0=1x(t0)⌘•p(ex|x)⌫ex2xx•rWl(f(x(t)))W•p(x|µ)µ•µh(x)•l(f(x))=logp(x|µ)•p(x|µ)=1(2⇡)D/2exp(12Pk(xkµk)2)µ=c+W⇤h(x)•AA=U⌃V>U·,k⌃k,kV>·,kB•B⇤=argminBs.t.rank(B)=k||AB||FB⇤=U·,k⌃k,kV>·,kargmin✓Xt12Xk(x(t)kbx(t)k)2argminW⇤,h(X)||XW⇤h(X)||F•x(t)X=U⌃V>argminW⇤,h(X)||XW⇤h(X)||F=W⇤ U·,k⌃k,k,h(X) V>·,kh(X)=V>·,k=V>·,k(X>X)1(X>X)=V>·,k(V⌃>U>U⌃V>)1(V⌃>U>X)=V>·,k(V⌃>⌃V>)1V⌃>U>X=V>·,kV(⌃>⌃)1V>V⌃>U>X=V>·,kV(⌃>⌃)1⌃>U>X=Ik,·(⌃>⌃)1⌃>U>X=Ik,·⌃1(⌃>)1⌃>U>X=Ik,·⌃1U>X=⌃1k,k(U·,k)>X•bx=(U·,k⌃k,k)h(x)h(x)=⇣⌃1k,k(U·,k)>⌘xW⇤W•x(t) 1pT⇣x(t)1TPTt0=1x(t0)⌘•p(ex|x)⌫ex2OriginalCorrupted(p=0.5)ReconstructedFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders6/36OriginalCorrupted(p=0.9)ReconstructedFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders7/36OriginalCorrupted(σ=2)ReconstructedFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders8/36OriginalCorrupted(σ=4)ReconstructedFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders9/36OriginalCorrupted(10×10)ReconstructedFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders10/36OriginalCorrupted(16×16)ReconstructedFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders11/36AkeyweaknessofthistypeofdenoisingisthattheposteriorµX|˜Xmaybenon-deterministic,possiblymulti-modal.Ifwetrainanautoencoderwiththequadraticloss,thebestreconstructionisφ(˜X)=EhX(cid:12)(cid:12)(cid:12)˜Xi,whichmaybeveryunlikelyunderµX|˜X.µX|˜XEhX(cid:12)(cid:12)(cid:12)˜XiFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders12/36ThisiswhatweobservewithverycorruptedMNISTdigits.X˜XModsofX|˜Xφ(˜X)’EhX(cid:12)(cid:12)(cid:12)˜XiFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders13/36Thiscanbemitigatedbyusinginplaceoflossasecondnetworkthatassessesiftheoutputisrealistic.Suchmethodsarecalledadversarialsincethesecondnetworkaimsatspottingthemistakesofthefirst,andthefirstaimsatfoolingthesecond.ItcanbecombinedwithastochasticdenoiserthatsamplesanXaccordingtoX|˜Xinsteadofcomputingadeterministicreconstruction.Wewillcomebacktothat.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders14/36Denoisingcanbeachievedwithoutcleansamples,ifthenoiseisadditiveandunbiased.Consider(cid:15)andδtwounbiasedandindependentnoises.WehaveEhkφ(X+(cid:15);θ)−(X+δ)k2i=Ehkφ(X+(cid:15);θ)−Xk2i−2Ehhδ,φ(X+(cid:15);θ)−Xii+Ehkδk2i=Ehkφ(X+(cid:15);θ)−Xk2i−2DE(cid:2)δ(cid:3),E(cid:2)φ(X+(cid:15);θ)−X(cid:3)E+Ehkδk2i=Ehkφ(X+(cid:15);θ)−Xk2i+Ehkδk2i.HenceargminθEhkφ(X+(cid:15);θ)−(X+δ)k2i=argminθEhkφ(X+(cid:15);θ)−Xk2i.UsingL1insteadofL2estimatesthemedianinsteadofthemean,andsimilarlyisstabletonoisethatkeepsthemedianunchanged.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders16/36Lehtinenetal.(2018)’sNoise2Noiseapproachusesthisforimagerestoration,asmanyexistingimagegenerativeprocessesinduceanunbiasednoise.Inmanyimagerestorationtasks,theexpectationofthecorruptedinputdataisthecleantargetthatweseektorestore.Low-lightphotographyisanexample:along,noise-freeexposureistheaverageofshort,independent,noisyexposures.PhysicallyaccuraterenderingsofvirtualenvironmentsaremostoftengeneratedthroughaprocessknownasMonteCarlopathtracing./.../TheMonteCarlointegratorisconstructedsuchthattheintensityofeachpixelistheexpectationoftherandompathsamplingprocess,i.e.,thesamplingnoiseiszero-mean.(Lehtinenetal.,2018)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders17/36Noise2Noise:LearningImageRestorationwithoutCleanDataA.AppendixA.1.NetworkarchitectureTable2showsthestructureoftheU-network(Ronnebergeretal.,2015)usedinallofourtests,withtheexceptionofthefirsttestinSection3.1thatusedthe“RED30”net-work(Maoetal.,2016).ForallbasicnoiseandtextremovalexperimentswithRGBimages,thenumberofinputandoutputchannelsweren=m=3.ForMonteCarlode-noisingwehadn=9,m=3,i.e.,inputcontainedRGBpixelcolor,RGBalbedo,anda3Dnormalvectorperpixel.TheMRIreconstructionwasdonewithmonochromeim-ages(n=m=1).Inputimageswererepresentedinrange[−0.5,0.5].A.2.TrainingparametersThenetworkweightswereinitializedfollowingHeetal.(2015).Nobatchnormalization,dropoutorotherreg-ularizationtechniqueswereused.Trainingwasdoneus-ingADAM(Kingma&Ba,2015)withparametervaluesβ1=0.9,β2=0.99,(cid:15)=10−8.Learningratewaskeptataconstantvalueduringtrainingexceptforabrieframpdownperiodatwhereitwassmoothlybroughttozero.Learningrateof0.001wasusedforallexperimentsexceptMonteCarlodenoising,where0.0003wasfoundtoprovidebetterstability.Minibatchsizeof4wasusedinallexperiments.A.3.FinitecorrupteddatainL2minimizationLetuscomputetheexpectederrorinL2normminimizationtaskwhencorruptedtargets{ˆyi}Ni=1areusedinplaceofthecleantargets{yi}Ni=1,withNafinitenumber.Letyibearbitraryrandomvariables,suchthatE{ˆyi}=yi.Asusual,thepointofleastdeviationisfoundattherespec-tivemean.Theexpectedsquareddifferencebetweenthesemeansacrossrealizationsofthenoiseisthen:Eˆy"1NXiyi−1NXiˆyi#2=1N2"Eˆy(Xiyi)2−2Eˆy"(Xiyi)(Xiˆyi)#+Eˆy(Xiˆyi)2#=1N2Var(Xiˆyi)=1N1NXiXjCov(ˆyi,ˆyj)(8)Intheintermediatesteps,wehaveusedEˆy(Piˆyi)=Piyiandbasicpropertiesof(co)variance.IfthecorruptionsareNAMENoutFUNCTIONINPUTnENCCONV048Convolution3×3ENCCONV148Convolution3×3POOL148Maxpool2×2ENCCONV248Convolution3×3POOL248Maxpool2×2ENCCONV348Convolution3×3POOL348Maxpool2×2ENCCONV448Convolution3×3POOL448Maxpool2×2ENCCONV548Convolution3×3POOL548Maxpool2×2ENCCONV648Convolution3×3UPSAMPLE548Upsample2×2CONCAT596ConcatenateoutputofPOOL4DECCONV5A96Convolution3×3DECCONV5B96Convolution3×3UPSAMPLE496Upsample2×2CONCAT4144ConcatenateoutputofPOOL3DECCONV4A96Convolution3×3DECCONV4B96Convolution3×3UPSAMPLE396Upsample2×2CONCAT3144ConcatenateoutputofPOOL2DECCONV3A96Convolution3×3DECCONV3B96Convolution3×3UPSAMPLE296Upsample2×2CONCAT2144ConcatenateoutputofPOOL1DECCONV2A96Convolution3×3DECCONV2B96Convolution3×3UPSAMPLE196Upsample2×2CONCAT196+nConcatenateINPUTDECCONV1A64Convolution3×3DECCONV1B32Convolution3×3DEVCONV1CmConvolution3×3,linearact.Table2.Networkarchitectureusedinourexperiments.Noutde-notesthenumberofoutputfeaturemapsforeachlayer.Numberofnetworkinputchannelsnandoutputchannelsmdependontheexperiment.Allconvolutionsusepaddingmode“same”,andexceptforthelastlayerarefollowedbyleakyReLUactivationfunction(Maasetal.,2013)withα=0.1.Otherlayershavelinearactivation.Upsamplingisnearest-neighbor.mutuallyuncorrelated,thelastrowsimplifiesto1N"1NXiVar(yi)#(9)Ineithercase,thevarianceoftheestimateistheaverage(co)varianceofthecorruptions,dividedbythenumberofsamplesN.Therefore,theerrorapproacheszeroasthenumberofsamplesgrows.Theestimateisunbiasedinthe(Lehtinenetal.,2018)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders18/36Noise2Noise:LearningImageRestorationwithoutCleanDataInput,8sppL2withˆx,ˆyL2withT(ˆx),ˆyL2withT(ˆx),T(ˆy)LHDRwithˆx,ˆyLHDRwithT(ˆx),ˆyReference,32kspp11.32dB25.46dB25.39dB15.50dB29.05dB30.09dBPSNRFigure6.ComparisonofvariouslossfunctionsfortrainingaMonteCarlodenoiserwithnoisytargetimagesrenderedat8samplesperpixel(spp).Inthishigh-dynamicrangesetting,ourcustomrelativelossLHDRisclearlysuperiortoL2.Applyinganon-lineartonemaptotheinputsisbeneficial,whileapplyingittothetargetimagesskewsthedistributionofnoiseandleadstowrong,visiblytoodarkresults.(a)Input(64spp),23.93dB(b)Noisytargets,32.42dB(c)Cleantargets,32.95dB(d)Reference(131kspp)Figure7.DenoisingaMonteCarlorenderedimage.(a)Imagerenderedwith64samplesperpixel.(b)Denoised64sppinput,trainedusing64spptargets.(c)Sameasprevious,buttrainedoncleantargets.(d)Referenceimagerenderedwith131072samplesperpixel.PSNRvaluesrefertotheimagesshownhere,seetextforaveragesovertheentirevalidationset.01020304001002003004005006007008009001000PSNRNoisy targetsClean targetsInputFigure8.OnlinetrainingPSNRduringa1000-frameflythroughofthesceneinFigure6.Noisytargetimagesarealmostasgoodforlearningascleantargets,butareover2000×fastertorender(190millisecondsvs7minutesperframeinthisscene).Bothdenoisersofferasubstantialimprovementoverthenoisyinput.prising,sincethetrainingdatasetcontainedonlyalimitednumberoftrainingpairs(andthusnoiserealizations)duetothecostofgeneratingthecleantargetimages,andwewantedtotestbothmethodsusingmatchingdata.Thatsaid,giventhatnoisytargetsare2000timesfastertopro-duce,onecouldtriviallyproducealargerquantityofthemandstillrealizevastgains.Thefinitecapturebudgetstudy(Section3.1)supportsthishypothesis.OnlinetrainingSinceitcanbetedioustocollectasuf-ficientlylargecorpusofMonteCarloimagesfortrainingagenerallyapplicabledenoiser,apossibilityistotrainamodelspecifictoasingle3Dscene,e.g.,agameleveloramovieshot(Chaitanyaetal.,2017).Inthiscontext,itcanevenbedesirabletotrainon-the-flywhilewalkingthroughthescene.Inordertomaintaininteractiveframerates,wecanaffordonlyfewsamplesperpixel,andthusbothinputandtargetimageswillbeinherentlynoisy.Figure8showstheconvergenceplotsforanexperimentwherewetrainedadenoiserfromscratchforthedurationof1000framesinasceneflythrough.OnanNVIDIATitanVGPU,pathtracingasingle512×512pixelimagewith8spptook190ms,andwerenderedtwoimagestoactasinputandtarget.Asinglenetworktrainingiterationwitharandom256×256pixelcroptook11.25msandweperformedeightofthemperframe.Finally,wedenoisedbothrenderedimages,eachtaking15ms,andaveragedtheresulttoproducethefinalimageshowntotheuser.Rendering,trainingandinferencetook500ms/frame.Figure8showsthattrainingwithcleantargetsdoesnotperformappreciablybetterthannoisytargets.Asrenderingasinglecleanimagetakesapprox.7minutesinthisscene(resp.190msforanoisytarget),thequality/timetradeoffclearlyfavorsnoisytargets.3.4.MagneticResonanceImaging(MRI)MagneticResonanceImaging(MRI)producesvolumetricimagesofbiologicaltissuesessentiallybysamplingthe(Lehtinenetal.,2018)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders19/36Noise2Noise:LearningImageRestorationwithoutCleanDataFouriertransform(the“k-space”)ofthesignal.ModernMRItechniqueshavelongreliedoncompressedsensing(CS)tocheattheNyquist-Shannonlimit:theyundersamplek-space,andperformnon-linearreconstructionthatremovesaliasingbyexploitingthesparsityoftheimageinasuitabletransformdomain(Lustigetal.,2008).Weobservethatifweturnthek-spacesamplingintoaran-domprocesswithaknownprobabilitydensityp(k)overthefrequenciesk,ourmainideaapplies.Inparticular,wemodelthek-spacesamplingoperationasaBernoulliprocesswhereeachindividualfrequencyhasaprobabilityp(k)=e−λ|k|ofbeingselectedforacquisition.4Thefrequenciesthatareretainedareweightedbytheinverseoftheselectionproba-bility,andnon-chosenfrequenciesaresettozero.Clearly,theexpectationofthis“Russianroulette”processisthecorrectspectrum.Theparameterλcontrolstheoverallfrac-tionofk-spaceretained;inthefollowingexperiments,wechooseitsothat10%ofthesamplesareretainedrelativetoafullNyquist-Shannonsampling.TheundersampledspectraaretransformedtotheprimalimagedomainbythestandardinverseFouriertransform.Anexampleofanundersam-pledinput/targetpicture,thecorrespondingfullysampledreference,andtheirspectra,areshowninFigure9(a,d).Nowwesimplysetuparegressionproblemoftheform(6)andtrainaconvolutionalneuralnetworkusingpairsoftwoindependentundersampledimagesˆxandˆyofthesamevol-ume.Asthespectraoftheinputandtargetarecorrectonex-pectation,andtheFouriertransformislinear,weusetheL2loss.Additionally,weimprovetheresultslightlybyenforc-ingtheexactpreservationoffrequenciesthatarepresentintheinputimageˆxbyFouriertransformingtheresultfθ(ˆx),replacingthefrequencieswiththosefromtheinput,andtransformingbacktotheprimaldomainbeforecomputingtheloss:thefinallossreads(F−1(Rˆx(F(fθ(ˆx))))−ˆy)2,whereRdenotesthereplacementofnon-zerofrequenciesfromtheinput.Thisprocessistrainedend-to-end.Weperformexperimentson2DslicesextractedfromtheIXIbrainscanMRIdataset.5Tosimulatespectralsampling,wedrawrandomsamplesfromtheFFTofthe(alreadyre-constructed)imagesinthedataset.Hence,indeviationfromactualMRIsamples,ourdataisreal-valuedandhastheperiodicityofthediscreteFFTbuilt-in.Thetrainingsetcontained5000imagesin256×256resolutionfrom50sub-jects,andforvalidationwechose1000randomimagesfrom10differentsubjects.ThebaselinePSNRofthesparsely-sampledinputimageswas20.03dBwhenreconstructeddirectlyusingIFFT.Thenetworktrainedfor300epochs4OursimplifiedexampledeviatesfrompracticalMRIinthesensethatwedonotsamplethespectraalong1Dtrajectories.However,webelievethatdesigningpulsesequencesthatleadtosimilarpseudo-randomsamplingcharacteristicsisstraightforward.5http://brain-development.org/ixi-dataset→T1images.ImageSpectrum(a)Input(b)Noisytrg.(c)Cleantrg.(d)Reference18.93dB29.77dB29.81dBFigure9.MRIreconstructionexample.(a)Inputimagewithonly10%ofspectrumsamplesretainedandscaledby1/p.(b)Recon-structionbyanetworktrainedwithnoisytargetimagessimilartotheinputimage.(c)Sameasprevious,buttrainingdonewithcleantargetimagessimilartothereferenceimage.(d)Original,uncorruptedimage.PSNRvaluesrefertotheimagesshownhere,seetextforaveragesovertheentirevalidationset.withnoisytargetsreachedanaveragePSNRof31.74dBonthevalidationdata,andthenetworktrainedwithcleantargetsreached31.77dB.Herethetrainingwithcleantar-getsissimilartopriorart(Wangetal.,2016;Leeetal.,2017).Trainingtook13hoursonanNVIDIATeslaP100GPU.Figure9(b,c)showsanexampleofreconstructionre-sultsbetweenconvolutionalnetworkstrainedwithnoisyandcleantargets,respectively.IntermsofPSNR,ourresultsquitecloselymatchthosereportedinrecentwork.4.DiscussionWehaveshownthatsimplestatisticalargumentsleadtonewcapabilitiesinlearnedsignalrecoveryusingdeepneuralnetworks;itispossibletorecoversignalsundercomplexcorruptionswithoutobservingcleansignals,withoutanexplicitstatisticalcharacterizationofthenoiseorothercor-ruption,atperformancelevelsequalorclosetousingcleantargetdata.Thatcleandataisnotnecessaryfordenoisingisnotanewobservation:indeed,consider,forinstance,theclassicBM3Dalgorithm(Dabovetal.,2007)thatdrawsonself-similarpatcheswithinasinglenoisyimage.Weshowthatthepreviously-demonstratedhighrestorationper-formanceofdeepneuralnetworkscanlikewisebeachievedentirelywithoutcleandata,allbasedonthesamegeneral-purposedeepconvolutionalmodel.Thispointsthewaytosignificantbenefitsinmanyapplicationsbyremovingtheneedforpotentiallystrenuouscollectionofcleandata.AmbientGAN(AshishBora,2018)trainsgenerativeadver-sarialnetworks(Goodfellowetal.,2014)usingcorruptedobservations.Incontrasttoourapproach,AmbientGANneedsanexplicitforwardmodelofthecorruption.Wefindcombiningideasalongbothpathsintriguing.(Lehtinenetal.,2018)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders20/36Super-resolutionFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders21/36OriginalInputAutoencoderoutputFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders26/36Limetal.(2017)usetwodifferentresnets.EDSRMDSRNbblocks3280Channels25664Nbparameters43M8Mimg034fromUrban100[10]HR(PSNR/SSIM)Bicubic(21.41dB/0.4810)A+[27](22.21dB/0.5408)SRCNN[4](22.33dB/0.5461)VDSR[11](22.62dB/0.5657)SRResNet[14](23.14dB/0.5891)EDSR+(Ours)(23.48dB/0.6048)MDSR+(Ours)(23.46dB/0.6039)img062fromUrban100[10]HR(PSNR/SSIM)Bicubic(19.82dB/0.6471)A+[27](20.43dB0.7145)SRCNN[4](20.61dB/0.7218)VDSR[11](20.75dB/0.7504)SRResNet[14](21.70dB/0.8054)EDSR+(Ours)(22.70dB/0.8537)MDSR+(Ours)(22.66dB/0.8508)0869fromDIV2K[26]HR(PSNR/SSIM)Bicubic(22.66dB/0.8025)A+[27](23.10dB/0.8251)SRCNN[4](23.14dB/0.8280)VDSR[11](23.36dB/0.8365)SRResNet[14](23.71dB/0.8485)EDSR+(Ours)(23.89dB/0.8563)MDSR+(Ours)(23.90dB/0.8558)Figure6:Qualitativecomparisonofourmodelswithotherworkson×4super-resolution.(Limetal.,2017)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders27/36Limetal.(2017)usetwodifferentresnets.EDSRMDSRNbblocks3280Channels25664Nbparameters43M8Mimg034fromUrban100[10]HR(PSNR/SSIM)Bicubic(21.41dB/0.4810)A+[27](22.21dB/0.5408)SRCNN[4](22.33dB/0.5461)VDSR[11](22.62dB/0.5657)SRResNet[14](23.14dB/0.5891)EDSR+(Ours)(23.48dB/0.6048)MDSR+(Ours)(23.46dB/0.6039)img062fromUrban100[10]HR(PSNR/SSIM)Bicubic(19.82dB/0.6471)A+[27](20.43dB0.7145)SRCNN[4](20.61dB/0.7218)VDSR[11](20.75dB/0.7504)SRResNet[14](21.70dB/0.8054)EDSR+(Ours)(22.70dB/0.8537)MDSR+(Ours)(22.66dB/0.8508)0869fromDIV2K[26]HR(PSNR/SSIM)Bicubic(22.66dB/0.8025)A+[27](23.10dB/0.8251)SRCNN[4](23.14dB/0.8280)VDSR[11](23.36dB/0.8365)SRResNet[14](23.71dB/0.8485)EDSR+(Ours)(23.89dB/0.8563)MDSR+(Ours)(23.90dB/0.8558)Figure6:Qualitativecomparisonofourmodelswithotherworkson×4super-resolution.(Limetal.,2017)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders27/36Limetal.(2017)usetwodifferentresnets.EDSRMDSRNbblocks3280Channels25664Nbparameters43M8Mimg034fromUrban100[10]HR(PSNR/SSIM)Bicubic(21.41dB/0.4810)A+[27](22.21dB/0.5408)SRCNN[4](22.33dB/0.5461)VDSR[11](22.62dB/0.5657)SRResNet[14](23.14dB/0.5891)EDSR+(Ours)(23.48dB/0.6048)MDSR+(Ours)(23.46dB/0.6039)img062fromUrban100[10]HR(PSNR/SSIM)Bicubic(19.82dB/0.6471)A+[27](20.43dB0.7145)SRCNN[4](20.61dB/0.7218)VDSR[11](20.75dB/0.7504)SRResNet[14](21.70dB/0.8054)EDSR+(Ours)(22.70dB/0.8537)MDSR+(Ours)(22.66dB/0.8508)0869fromDIV2K[26]HR(PSNR/SSIM)Bicubic(22.66dB/0.8025)A+[27](23.10dB/0.8251)SRCNN[4](23.14dB/0.8280)VDSR[11](23.36dB/0.8365)SRResNet[14](23.71dB/0.8485)EDSR+(Ours)(23.89dB/0.8563)MDSR+(Ours)(23.90dB/0.8558)Figure6:Qualitativecomparisonofourmodelswithotherworkson×4super-resolution.(Limetal.,2017)Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders27/36Autoencodersasself-trainingFran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders28/36Vincentetal.(2010)interprettrainingtheautoencoderasmaximizingthemutualinformationbetweentheinputandthelatentstates.LetXbeasample,Z=f(X;θ)itslatentrepresentation,andq(x,z)thedistributionof(X,Z).WehaveargmaxθI(X;Z)=argmaxθEq(X,Z)hlogq(X|Z)i.However,thereisnoexpressionofq(X|Z)inanyreasonablesetup.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders29/36Vincentetal.builddeepMLPswhoselayersareinitializedsuccessivelyasencoderstrainedwithinanoisyautoencoder.AutoencoderAutoencoderAutoencoderAfinalclassifyinglayerisaddedandthefullstructurecanbefine-tuned.Thisapproach,andothersinthesamespirit(Hintonetal.,2006),wereseenasstrategiestocomplementgradient-descentforbuildingdeepnets.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders35/36Vincentetal.builddeepMLPswhoselayersareinitializedsuccessivelyasencoderstrainedwithinanoisyautoencoder.AutoencoderAutoencoderAutoencoderAfinalclassifyinglayerisaddedandthefullstructurecanbefine-tuned.Thisapproach,andothersinthesamespirit(Hintonetal.,2006),wereseenasstrategiestocomplementgradient-descentforbuildingdeepnets.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders35/36Vincentetal.builddeepMLPswhoselayersareinitializedsuccessivelyasencoderstrainedwithinanoisyautoencoder.AutoencoderAutoencoderAutoencoderAfinalclassifyinglayerisaddedandthefullstructurecanbefine-tuned.Thisapproach,andothersinthesamespirit(Hintonetal.,2006),wereseenasstrategiestocomplementgradient-descentforbuildingdeepnets.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders35/36Vincentetal.builddeepMLPswhoselayersareinitializedsuccessivelyasencoderstrainedwithinanoisyautoencoder.AutoencoderAutoencoderAutoencoderAfinalclassifyinglayerisaddedandthefullstructurecanbefine-tuned.Thisapproach,andothersinthesamespirit(Hintonetal.,2006),wereseenasstrategiestocomplementgradient-descentforbuildingdeepnets.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders35/36Vincentetal.builddeepMLPswhoselayersareinitializedsuccessivelyasencoderstrainedwithinanoisyautoencoder.AutoencoderAutoencoderAutoencoderAfinalclassifyinglayerisaddedandthefullstructurecanbefine-tuned.Thisapproach,andothersinthesamespirit(Hintonetal.,2006),wereseenasstrategiestocomplementgradient-descentforbuildingdeepnets.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders35/36Vincentetal.builddeepMLPswhoselayersareinitializedsuccessivelyasencoderstrainedwithinanoisyautoencoder.AutoencoderAutoencoderAutoencoderAfinalclassifyinglayerisaddedandthefullstructurecanbefine-tuned.Thisapproach,andothersinthesamespirit(Hintonetal.,2006),wereseenasstrategiestocomplementgradient-descentforbuildingdeepnets.Fran¸coisFleuretEE-559–Deeplearning/7.3.Denoisingautoencoders35/36Variational AutoEncoder (VAE)

Weconsiderthefollowingtwodistributions:•pisthedistributionon𝒳×Rdofapair(X,Z)composedofanencodingstateZ∼𝒩(0,I)andtheoutputofthedecodergonit.•qisthedistributionon𝒳×Rdofapair(X,Z)composedofasampleXtakenfromthedatadistributionandtheoutputoftheencoderonit,Ourgoalisthatp(X)mimicsthedata-distributionq(X),thatistofindgthatmaximizesthelog-likelihood1NXnlogp(xn)=ˆEq(X)hlogp(X)i.However,withacomplicatedg,wecansamplezandcomputeg(z),butcannotcomputep(x)foragivenx,andevenlesscomputeitsderivatives.Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders2/15Weconsiderthefollowingtwodistributions:•pisthedistributionon𝒳×Rdofapair(X,Z)composedofanencodingstateZ∼𝒩(0,I)andtheoutputofthedecodergonit.•qisthedistributionon𝒳×Rdofapair(X,Z)composedofasampleXtakenfromthedatadistributionandtheoutputoftheencoderonit,Ourgoalisthatp(X)mimicsthedata-distributionq(X),thatistofindgthatmaximizesthelog-likelihood1NXnlogp(xn)=ˆEq(X)hlogp(X)i.However,withacomplicatedg,wecansamplezandcomputeg(z),butcannotcomputep(x)foragivenx,andevenlesscomputeitsderivatives.Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders2/15TheVariationalAutoencoderproposedbyKingmaandWelling(2013)reliesonatractableapproximationofthislog-likelihood.Notethattheirframeworkinvolvesstochasticencoderf,anddecoderg,whoseoutputsdependonboththeirinputsandadditionalrandomness.Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders3/15Rememberthatq(X)isthedatadistribution,andq(Z|X=x)isthedistributionofthelatentencodingf(x).WewanttomaximizeEq(X)hlogp(X)i,andwecanshowthat−Eq(X)hlogp(X)i≤Eq(X)hDKL(q(Z|X)kp(Z))i−Eq(X,Z)hlogp(X|Z)i.Soitmakessensetominimizethislatterquantity.Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders4/15Sothefinallossisℒ=Eq(X)hDKL(q(Z|X)kp(Z))i−Eq(X,Z)hlogp(X|Z)i.with•q(X)isthedatadistribution•p(Z)=𝒩(0,I).KingmaandWellingproposethatboththeencoderfanddecodergmaptoaGaussianwithdiagonalcovariance.Hencetheymaptotwicethedimension(e.g.f(x)=(µf(x),σf(x)))and•q(Z|X=x)∼𝒩(µf(x),diag(σf(x)))•p(X|Z=z)∼𝒩(µg(z),diag(σg(z))).Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders5/15ThefirsttermofℒistheaverageofDKL(cid:0)q(Z|X=x)|{z}𝒩(µf(x),σf(x))kp(Z)|{z}𝒩(0,I)(cid:1)=−12Xd(cid:18)1+2logσfd(x)−(cid:16)µfd(x)(cid:17)2−(cid:16)σfd(x)(cid:17)2(cid:19).overthexns.Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders6/15ThefirsttermofℒistheaverageofDKL(cid:0)q(Z|X=x)|{z}𝒩(µf(x),σf(x))kp(Z)|{z}𝒩(0,I)(cid:1)=−12Xd(cid:18)1+2logσfd(x)−(cid:16)µfd(x)(cid:17)2−(cid:16)σfd(x)(cid:17)2(cid:19).overthexns.Originalspace𝒳LatentspaceℱfFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders6/15ThefirsttermofℒistheaverageofDKL(cid:0)q(Z|X=x)|{z}𝒩(µf(x),σf(x))kp(Z)|{z}𝒩(0,I)(cid:1)=−12Xd(cid:18)1+2logσfd(x)−(cid:16)µfd(x)(cid:17)2−(cid:16)σfd(x)(cid:17)2(cid:19).overthexns.Originalspace𝒳LatentspaceℱfFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders6/15ThefirsttermofℒistheaverageofDKL(cid:0)q(Z|X=x)|{z}𝒩(µf(x),σf(x))kp(Z)|{z}𝒩(0,I)(cid:1)=−12Xd(cid:18)1+2logσfd(x)−(cid:16)µfd(x)(cid:17)2−(cid:16)σfd(x)(cid:17)2(cid:19).overthexns.Originalspace𝒳LatentspaceℱfFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders6/15ThefirsttermofℒistheaverageofDKL(cid:0)q(Z|X=x)|{z}𝒩(µf(x),σf(x))kp(Z)|{z}𝒩(0,I)(cid:1)=−12Xd(cid:18)1+2logσfd(x)−(cid:16)µfd(x)(cid:17)2−(cid:16)σfd(x)(cid:17)2(cid:19).overthexns.Originalspace𝒳LatentspaceℱfFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders6/15AsKingmaandWelling(2013),weuseaconstantvarianceof1forthedecoder,sothesecondtermofℒbecomestheaverageof−logp(X=x|Z=z)=12Xd(xd−µgd(z))2+cstoverthexn,withoneznsampledforeach,i.e.zn∼𝒩(cid:16)µf(xn),σf(xn)(cid:17),n=1,...,N.Originalspace𝒳LatentspaceℱfgFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders7/15AsKingmaandWelling(2013),weuseaconstantvarianceof1forthedecoder,sothesecondtermofℒbecomestheaverageof−logp(X=x|Z=z)=12Xd(xd−µgd(z))2+cstoverthexn,withoneznsampledforeach,i.e.zn∼𝒩(cid:16)µf(xn),σf(xn)(cid:17),n=1,...,N.Originalspace𝒳LatentspaceℱfgFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders7/15AsKingmaandWelling(2013),weuseaconstantvarianceof1forthedecoder,sothesecondtermofℒbecomestheaverageof−logp(X=x|Z=z)=12Xd(xd−µgd(z))2+cstoverthexn,withoneznsampledforeach,i.e.zn∼𝒩(cid:16)µf(xn),σf(xn)(cid:17),n=1,...,N.Originalspace𝒳LatentspaceℱfgFran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders7/15OriginalAutoencoderreconstruction(d=32)VariationalAutoencoderreconstruction(d=32)Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders10/15Autoencodersampling(d=32)VariationalAutoencodersampling(d=32)Fran¸coisFleuretEE-559–Deeplearning/7.4.Variationalautoencoders12/15Context Encoders: Feature Learning by Inpainting, Pathak et. al., 2016Context encoderContext encoderSegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, Badrinarayanan et. al., 2016Link (note: this is supervised learning)SegNetA small overview of

applications in computer Vision

We use the slides of C. Wolf

Image classificationObject detectionand recognitionSemanticsegmentationInstance segmentation1234AlexNetThe model whichmade deeplearninghugelypopularby winningthe Imagenetcompetitionin 2012.8 trainablelayers(5 convolutions, 3 fullyconnected).Contribution: ReLUactivation, dropout, multi-GPU training[Krizhevsky, Sutskever, Hinton, 2012]VGG 16[Simonyanand Zissermann, ICLR 2015]-16 layers-Only3x3 convolutionsResNetWinsImageNet2015 competition.Novelty: residualblocks –a block predictsthe differenceto itsinput.[He, Zhang, Ren, Sun, CVPR 2016]Resnet 34ResNet: variants[He, Zhang, Ren, Sun, CVPR 2016]Resnet 34Resnet 50/101/152Image classificationObject detectionand recognitionSemanticsegmentationInstance segmentation1234Detection, localization, recognitionA more complexproblem. Weneedto:-Detectwhetheran objectexists-Localizeit(regressitsboundingbox coordinates)-RecognizeitsclassMultiple instances are possible[Figure: FasterR-CNN, Ren, He, Girshick, Sun, NIPS 2015]BeforeDeepLearning: DeformableParts Models-Model an object/human/activityas a collection of local parts-Learna filterfor eachpart-Learnan anchorposition and deformationcoefficeientsfor eachpart-Test eachimage pixel whetheritcanbethe center of the object(slidingwindow):-For eachpossible center, optimizeover (latent) local part positions[Felzenszwalbet al., PAMI 2010]Local appearanceDeformationSpatial feature mapsMost vision methods requiring localization exploit the fact that in convolutional neural networks (CNNs) intermediate layer activations have a spatial meaning: each cell corresponds to a rectangular area in the input image (“receptive field”).[Figure: Damien Fourure, PhD thesis, 2017]R-CNN-Detecta large numberof candidate regions(«regionproposals») withsomeheuristicmethod-Feedeachcandidate regionintoa convolutionalneural network for recognition[Girshick, Donahue, Darrel, Malik, 2014]Faster R-CNN-The region proposals are now predicted by a neural network which is part of the full network-For each proposal, features are collected from the bounding box of the proposaland fed to a classifier. [Ren, He, Girshick, Sun, NIPS 2015][Erhanet al.,2014]«Multibox»Real time detectors[Redmonet al.,2016]«YOLO»[Erhanet al.,2016]«SSD»Image classificationObject detectionand recognitionSemanticsegmentationInstance segmentation1234[Fourureet al., BMVC 2017]Recall: receptive Fields[Figure: Damien Fourure, PhD thesis, 2017]Pooling layers reduce the spatial resolution of intermediate activations and output layers.How can we create dense predictions while keeping the output resolution equal to the input resolution?Patchwiseprocessing[Figure: Damien Fourure, PhD thesis, 2017]A direct, simple and earlymethod. Not usedanymoreFeedeachpixel + neighborhoodas input intoa network. Slow!Conv-DeconvNetworks[Nohet al., 2015]A classical ”encoder network” produces a vectorialrepresentation through convolutions and pooling.A second network (“decoder”) decodes this into an output image with the initial resolution.Allinformationmustpassthroughthebottlenecklayer!Conv-DeconvNetworks[Nohet al., 2015]In the decoder:-convolutions are replaced with “deconvolutions” or “transposed convolutions”.-Poolingisreplacedwithunpooling(switch variables keep the argmax location of the pooling layer)U-Nets[Ronnebergeret al., MICCAI 2015]Conv-Deconv: all information passes through the bottleneck layer.U-Nets add additional skip connections for low level information, close to pixels.DilatedConvolutionsAlternative: do not use any pooling, keep spatial resolution throughout the network.To increase the receptive field, change the size of the filters …w/o augmenting the number of parameters![Yuand Koltun, ICLR 2016]GridNetworksGridNetworks generalizea large numberof networks.[Fourure, Emonet, Fromont, Muselet, Tremeau, Wolf, BMVC 2017][Fourure, Emonet, Fromont, Muselet, Tremeau, Wolf, BMVC 2017]Image classificationObject detectionand recognitionSemanticsegmentationInstance segmentation1234Instance segmentationBridges the gap betweensemanticsegmentation and objectdetection:-One region= 1 objectinstance-Pixelwiseboundariesinsteadof boundingboxes[Figure: Arnabet al., IEEE Signal Processing, 2018]MaskR-CNNRegionproposalsasinobjectdetectionConv-Deconvnetwork likein semanticsegmentation ... but for eachregionproposal.ResNet-101 Backbone.[Hen, Gkioxkari, Dollar, Girshick, ICCV 2017]MaskR-CNN[Hen, Gkioxkari, Dollar, Girshick, ICCV 2017]Dense Poseinputoutput[Güler, Neverova, Kokkinos, CVPR 2018]Slides: Natalia NeverovaDenseposeestimateshumanarticulatedpose in a dense manner:One estimateper image...