Standard Notations for Deep Learning

·ˆy ∈ Rny is the predicted output vector. It can also be denoted a[L] where L

denote the lth layer

Sizes:

·m : number of examples in the dataset

·nx : input size

·ny : output size (or number of classes)

·n[l]

h : number of hidden units of the lth layer

is the number of layers in the network.

Common forward propagation equation examples:

a = g[l](Wxx(i) + b1) = g[l](z1) where g[l] denotes the lth layer activation

function

ˆy(i) = sof tmax(Whh + b2)

· General Activation Formula: a[l]

j = g[l]((cid:80)

k w[l]

jka[l−1]

k

+ b[l]

j ) = g[l](z[l]

j )

In a for loop, it is possible to denote nx = n[0]

h and ny = nh

[number of layers +1].

· J(x, W, b, y) or J(ˆy, y) denote the cost function.

·L : number of layers in the network.

Objects:

·X ∈ Rnx×m is the input matrix

·x(i) ∈ Rnx is the ithexample represented as a column vector

Examples of cost function:

· JCE(ˆy, y) = − (cid:80)m

i=0 y(i) log ˆy(i)

· J1(ˆy, y) = (cid:80)m

i=0 | y(i) − ˆy(i) |

1

2 Deep Learning representations

For representations:

· nodes represent inputs, activations or outputs

· edges represent weights or biases

Here are several examples of Standard deep learning representations

2

Figure 1: Comprehensive Network: representation commonly used for Neural

Networks. For better aesthetic, we omitted the details on the parameters (w[l]

ij

and b[l]

i etc...) that should appear on the edges

Figure 2: Simplified Network: a simpler representation of a two layer neural

network, both are equivalent.