Single-Layer Perceptron



Notes:

First, let's take a look at the simplest neural network — single-layer perceptron.

In 1958, Frank Rosenblatt invented the perceptron algorithm. Such an algorithm coexisted with machine learning for more than a decade.

In 1969, Marvin Minsky, pioneer of artificial intelligence in the US, questioned that the perceptron could handle only linear classification problems and failed to classify even the simplest XOR problems. The research on the perceptron was doomed to failure.

<!-- Slide number: 12 -->

XOR Problem

In 1969, Minsky, an American mathematician and AI pioneer, proved that a perceptron is essentially a linear model that can only deal with linear classification problems, but cannot process non-linear data.

AND

OR

XOR

Notes:

Restriction of the single-layer perceptron: XOR problem

Reference: https://blog.csdn.net/qq_18515405/article/details/42123697

Under this context, the multilayer perceptron was invented.

<!-- Slide number: 13 -->

Feedforward Neural Network

Output layer

Input layer

Hidden layer 1

Hidden layer 2

Notes:

The multilayer perceptron is a feedforward neural network.

It is the simplest neural network with neurons arranged in layers. It is more widely used and develops faster than other artificial neural networks.

Input nodes do not have the calculation capability and are only used to represent element values of the input vector.

The neuron having the computing capability at each layer is referred to as the computing unit. Each neuron is connected only to the neurons of the previous layer.

A unidirectional multi-layer structure is used to receive the output of the previous layer and send the output to the next layer. Each layer includes several neurons, and the neurons at the same layer are not connected to each other. Only unidirectional inter-layer information transmission is supported.

<!-- Slide number: 14 -->

Solution of XOR

w0

XOR

XOR

w1

w2

w3

w4

w5

Notes:

<!-- Slide number: 15 -->

Impacts of Hidden Layers on A Neural Network




0 hidden layers

3 hidden layers

20 hidden layers

Notes:

More hidden layers indicate the stronger identification capability of the neural network.

<!-- Slide number: 16 -->

Deep Learning Summary

Training Rules

Activation Function

Normalizer

Optimizer

Types of Neural Networks

Common Problems

Notes:

<!-- Slide number: 17 -->

Gradient Descent and Loss Function

Notes:

<!-- Slide number: 18 -->

Extrema of the Loss Function


Example of gradient descent of binary paraboloid

Notes:

<!-- Slide number: 19 -->

Common Loss Functions in Deep Learning

Notes:

<!-- Slide number: 20 -->

Batch Gradient Descent Algorithm (BGD)

Notes:

<!-- Slide number: 21 -->

Stochastic Gradient Descent Algorithm (SGD)

Notes:

This gradient descent algorithm goes to another extreme, that is, updating the weight based on each sample. Most training samples contain noises. As a result, when the extrema is approximated to, the gradient direction is oriented up and down near the extrema but difficult to converge to the extrema.

<!-- Slide number: 22 -->

Mini-Batch Gradient Descent Algorithm (MBGD)

Notes:

This gradient descent algorithm considers both the efficiency and gradient stability. It is easy to overshoot the local minimum and is the most commonly used gradient descent algorithm in actual work. The value of BS varies with specific problems. Generally, the value is 128.

<!-- Slide number: 23 -->

Backpropagation Algorithm (1)

Signals are propagated in forward direction, and errors are propagated in backward direction.

In the training sample set D, each sample is recorded as <X, t>, in which X is the input vector, t the target output, o the actual output, and w the weight coefficient.

Loss function:

Forward propagation direction

Output layer

Input layer

Hidden layer

Backpropagation direction

Notes:

<!-- Slide number: 24 -->

Backpropagation Algorithm (2)

Output layer error:

Expanded hidden layer error:

Expanded input layer error:

Notes:

<!-- Slide number: 25 -->

Backpropagation Algorithm (3)

Notes:

<!-- Slide number: 26 -->

Backpropagation Algorithm (4)

For a neural network with any number of layers, the arranged formula for training is as follows:

The BP algorithm is used to train the network as follows:

Takes out the next training sample <X, T>, inputs X to the network, and obtains the actual output o.

Calculates output layer δ according to the output layer error formula (1).

Calculates δ of each hidden layer from output to input by iteration according to the hidden layer error propagation formula (2).

According to the δ of each layer, the weight values of all the layer are updated.

Notes:

<!-- Slide number: 27 -->

Deep Learning Summary

Training Rules

Activation Function

Normalizer

Optimizer

Types of Neural Networks

Common Problems

Notes:

<!-- Slide number: 28 -->

Activation Function

Activation functions are important for the neural network model to learn and understand complex non-linear functions. They allow introduction of non-linear features to the network.

Without activation functions, output signals are only simple linear functions. The complexity of linear functions is limited, and the capability of learning complex function mappings from data is low.

Activation Function

Notes:

<!-- Slide number: 29 -->

Sigmoid


Notes:

The sigmoid function is monotonic, continuous, and easy to derive. The output is bounded, and the network is easy to converge. However, we see that the derivative of the sigmoid function is close to 0 at the position away from the central point. When the network is very deep, more and more backpropagation gradients fall into the saturation area so that the gradient module becomes smaller. Generally, if the sigmoid network has five or fewer layers, the gradient is degraded to 0, which is difficult to train. This phenomenon is a vanishing gradient. In addition, the output of the sigmoid is not zero-centered.

<!-- Slide number: 30 -->

Tanh


Notes:

Tanh function and sigmoid function have similar shortcomings. The derivative of the tanh function is nearly 0 at its extremes. However, because the tanh function is symmetric with respect to the origin, the average of the outputs is closer to 0 than that of the sigmoid function. Therefore, SGD can reduce the required number of iterations because it is closer to the natural gradient descent.

<!-- Slide number: 31 -->

Softsign


Notes:

This function saturates more slowly than the tanh function.

When the sigmoid, tanh, and softsign functions are used to train a deep neural network, the vanishing gradient problem is inevitable. The derivative of the functions approaches 0 at its extremes. When the network is very deep, more and more backpropagation gradients fall into the saturation area so that the gradient module becomes smaller and finally close to 0, and the weight cannot be updated.

Generally, if the neural network has more than five layers, the gradient is degraded to 0, which is difficult to train.

<!-- Slide number: 32 -->

Rectified Linear Unit (ReLU)


Notes:

Advantages:

Compared with sigmoid and tanh, ReLU supports fast convergence in SGD.

Compared with the sigmoid and tanh functions involving exponentiation, the ReLU can be implemented more easily.

The vanishing gradient problem can be effectively alleviated.

The ReLU has a good performance during unsupervised pre-training.

Disadvantages:

There is no upper bound, so that the training is relatively easy to diverge.

The ReLU is not differentiable at x = 0 and a derivative is forcibly defined at this point.

The surface defined at the zero point is not smooth enough in some regression problems.

Reduces the computation workload

When functions such as sigmoid are used, the activation function involves exponent operation, which requires a large amount of computation. When the error gradient is calculated through backpropagation, the derivation involves division and the computation workload is heavy. However, the ReLU activation function can reduce much of the computation workload.

Effectively mitigates the vanishing gradient problem.

The ReLU gradient is unsaturated. When the sigmoid function is close to the saturation area (far from the function center), the transformation is too slow and the derivative is close to 0. Therefore, in the backpropagation process, the ReLU function mitigates the vanishing gradient problem, and parameters of the first several layers of the neural network can be quickly updated.

<!-- Slide number: 33 -->

Softplus


Notes:

Compared with ReLU, this function has more complex computation. However, it has a continuous derivative and defines a smooth curved surface.

<!-- Slide number: 34 -->

Softmax

Notes:

<!-- Slide number: 35 -->

Deep Learning Summary

Training Rules

Activation Function

Normalizer

Optimizer

Types of Neural Networks

Common Problems

Notes:

<!-- Slide number: 36 -->

Normalizer

Notes:

<!-- Slide number: 37 -->

Penalty Parameters

Notes:

<!-- Slide number: 38 -->

Notes:

<!-- Slide number: 39 -->

Notes:

<!-- Slide number: 40 -->


Notes:

<!-- Slide number: 41 -->

Dataset Expansion

Notes:

<!-- Slide number: 42 -->

Dropout

Dropout is a common and simple regularization method, which has been widely used since 2014. Simply put, Dropout randomly discards some inputs during the training process. In this case, the parameters corresponding to the discarded inputs are not updated. As an integration method, Dropout combines all sub-network results and obtains sub-networks by randomly dropping inputs. See the figures below:



Dropout in training Testing

Notes:

The sampling probability of each entry is 0.8 for the input and 0.5 for the hidden layers.

Advantages:

Compared with weight decay and norm constraints, this strategy is more effective.

It is computationally cheap and simple and can be used in other non-deep-learning models.

However, it is less effective when the training data is insufficient.

Stochasticity is not necessary or sufficient to achieve the regularizing effect of dropout. Invariant shielding parameters can be constructed to obtain good solutions.

In addition to the preceding methods, we can also use semi-supervised learning, multi-task learning, early stopping, parameter sharing, ensemble methods, and adversarial training.

<!-- Slide number: 43 -->

Early Stopping

A test on data of the validation set can be inserted during the training. When the data loss of the verification set increases, perform early stopping.


Early stopping

Notes:

<!-- Slide number: 44 -->

Deep Learning Summary

Training Rules

Activation Function

Normalizer

Optimizer

Types of Neural Networks

Common Problems

Notes:

<!-- Slide number: 45 -->

Optimizer

There are various optimized versions of gradient descent algorithms. In object-oriented language implementation, different gradient descent algorithms are often encapsulated into objects called optimizers.

Purposes of the algorithm optimization include but are not limited to:

Accelerating algorithm convergence.

Preventing or jumping out of local extreme values.

Simplifying manual parameter setting, especially the learning rate (LR).

Common optimizers: common GD optimizer, momentum optimizer, Nesterov, AdaGrad, AdaDelta, RMSProp, Adam, AdaMax, and Nadam.

Notes:

<!-- Slide number: 46 -->

Momentum Optimizer


Notes:

<!-- Slide number: 47 -->

Advantages and Disadvantages of Momentum Optimizer


Notes:

<!-- Slide number: 48 -->

AdaGrad Optimizer (1)

Gradient calculation

Square gradient accumulation

Computing update

Application update

Notes:

<!-- Slide number: 49 -->

AdaGrad Optimizer (2)

Notes:

<!-- Slide number: 50 -->

RMSProp Optimizer

Gradient calculation

Square gradient accumulation

Computing update

Application update

Notes:

m indicates the number of samples used.

<!-- Slide number: 51 -->

Adam Optimizer (1)

Notes:

Adam attempts to calculate adaptive learning rates for each parameter. This is very useful in complex network structures because different parts of the network have different sensitivity to weight adjustment. A very sensitive part usually requires a smaller learning rate. It is difficult or complex to manually identify the sensitive part and set a learning rate. It may be the best optimizer at present.

<!-- Slide number: 52 -->

Adam Optimizer (2)

Notes:

<!-- Slide number: 53 -->

Optimizer Performance Comparison



Comparison of optimization algorithms in contour maps of loss functions

Comparison of optimization algorithms at the saddle point

Notes:

<!-- Slide number: 54 -->

Deep Learning Summary

Training Rules

Activation Function

Normalizer

Optimizer

Types of Neural Networks

Common Problems

Notes:

<!-- Slide number: 55 -->

Convolutional Neural Network

A convolutional neural network (CNN) is a feedforward neural network. Its artificial neurons may respond to surrounding units within the coverage range. CNN excels at image processing. It includes a convolutional layer, a pooling layer, and a fully connected layer.

In the 1960s, Hubel and Wiesel studied cats' cortex neurons used for local sensitivity and direction selection and found that their unique network structure could simplify feedback neural networks. They then proposed the CNN.

Now, CNN has become one of the research hotspots in many scientific fields, especially in the pattern classification field. The network is widely used because it can avoid complex pre-processing of images and directly input original images.

Notes:

A filter matrix is a set of fixed weight and can be seen as a constant filter (kernel). The convolution (adding each element, weighted by the kernel) is performed between an image (data from different data windows) and a kernel. This type of network is called CNN.

Local receptive field: It is generally considered that human perception of the outside world is from local to global. Spatial correlations among local pixels of an image are closer than those among pixels that are far away. Therefore, each neuron does not need to know the global image. It only needs to know the local image and then the local information is combined at a higher level to generate global information. The idea of local network connection is also inspired by the biological visual system structure. The neurons in the visual cortex receive local information (respond to stimuli of certain regions).

Parameter sharing: One or more filters can be used to scan the input images. The parameters of the filter are weights. At the layers scanned by the same filter, each filter uses the same parameters to perform weighted calculation. Weight sharing means that the parameter values of each filter does not change when the filter scans the entire image. For example, if we have three feature filters and each filter scans the entire image. During the scanning process, the parameter values of the filters do not change. In other words, all elements of the image share the same weights.

<!-- Slide number: 56 -->

Main Concepts of CNN

Local receptive field: It is generally considered that human perception of the outside world is from local to global. Spatial correlations among local pixels of an image are closer than those among distant pixels. Therefore, each neuron does not need to know the global image. It only needs to know the local image. The local information is combined at a higher level to generate global information.

Parameter sharing: One or more filters/kernels may be used to scan input images. Parameters carried by the filters are weights. In a layer scanned by filters, each filter uses the same parameters during weighted computation. Weight sharing means that when each filter scans an entire image, parameters of the filter are fixed.

Notes:

Local receptive field: The idea of local network connection is also inspired by the biological visual system structure. The neurons in the visual cortex receive local information (respond to stimuli of certain regions).

Parameter sharing: For example, if we have three feature convolution kernels and each kernel scans the entire image, the parameter values of the convolution kernels do not change during the scanning process. In other words, all elements of the image share the same weights. This means that the features learned from a part of the image can also be applied to other parts of the image or other images, which is called position invariance.

<!-- Slide number: 57 -->

Architecture of Convolutional Neural Network


Input image

Output layer

Three-feature image

Three-feature image

Five-feature image

Five-feature image

Pooling layer

Convolutional layer

Convolutional layer

Pooling layer

Fully connected layer


Pbird

Bird

Psunset

Sunset

Pdog

Dog

Pcat

Cat

Vectorization

Convolution + nonlinearity

Max pooling

Multi-category

Convolution layers + pooling layers

Fully connected layer

Notes:

Input layer: inputs data.

Convolutional layer: composed of several convolutional units. The parameters of each convolutional unit are obtained by optimizing the backpropagation algorithm. The purpose of convolution calculation is to extract different input features. The first convolutional layer may extract only some low-level features such as edges, lines, and angles. A multi-layer network can extract more complex features based on the low-level features.

Rectified linear units layer (ReLU layer): uses ReLU f(x) = max(0, x) as the activation function.

Pooling layer: partitions features obtained from the convolutional layer into some areas and outputs the maximum or minimum value, generating new features with a smaller spatial size.

Fully connected layer: integrates all local features into global features to calculate the final scores for each type.

Output layer: outputs the final result.

<!-- Slide number: 58 -->

Single-Filter Calculation (1)

Description of convolution calculation


Notes:

<!-- Slide number: 59 -->

Single-Filter Calculation (2)

Demonstration of the convolution calculation


Han Bingtao, 2017, Convolutional Neural Network

Notes:

<!-- Slide number: 60 -->

Convolutional Layer

The basic architecture of a CNN is multi-channel convolution consisting of multiple single convolutions. The output of the previous layer (or the original image of the first layer) is used as the input of the current layer. It is then convolved with the filter in the layer and serves as the output of this layer. The convolution kernel of each layer is the weight to be learned. Similar to FCN, after the convolution is complete, the result should be biased and activated through activation functions before being input to the next layer.


Output tensor

Input tensor

Activate

Output

Bias

Convolutional kernel

Notes:

<!-- Slide number: 61 -->

Pooling Layer

Pooling combines nearby units to reduce the size of the input on the next layer, reducing dimensions. Common pooling includes max pooling and average pooling. When max pooling is used, the maximum value in a small square area is selected as the representative of this area, while the mean value is selected as the representative when average pooling is used. The side of this small area is the pool window size. The following figure shows the max pooling operation whose pooling window size is 2.


Sliding direction

Notes:

The actual classification networks are feedforward networks that are formed by interconnected convolutional and pooling layers. The pooling layer has the following functions:

Invariance: Max pooling ensures invariance within a certain range, because the maximum value of a region is the last output value regardless of where the value is.

Reducing the input size for the next layer: Pooling effectively reduces the size of the input data for the next layer, the number of parameters, and computation workload.

Obtaining fixed-length data: By properly setting the pooling window size and stride, we can obtain fixed-length outputs from variable-length inputs.

Increasing the scale: The features of the previous layer can be extracted from a larger scale.

Preventing overfitting: Pooling simplifies the network and reduces the fitting precision. Therefore, it can prevent overfitting (pay attention to the possible underfitting).

<!-- Slide number: 62 -->

Fully Connected Layer

Notes:

<!-- Slide number: 63 -->

Recurrent Neural Network

The recurrent neural network (RNN) is a neural network that captures dynamic information in sequential data through periodical connections of hidden layer nodes. It can classify sequential data.

Unlike other forward neural networks, the RNN can keep a context state and even store, learn, and express related information in context windows of any length. Different from traditional neural networks, it is not limited to the space boundary, but also supports time sequences. In other words, there is a side between the hidden layer of the current moment and the hidden layer of the next moment.

The RNN is widely used in scenarios related to sequences, such as videos consisting of image frames, audio consisting of clips, and sentences consisting of words.

Notes:

<!-- Slide number: 64 -->

Recurrent Neural Network Architecture (1)


Notes:

<!-- Slide number: 65 -->

Recurrent Neural Network Architecture (2)


LeCun, Bengio, and G. Hinton, 2015, A Recurrent Neural Network and the

Unfolding in Time of the Computation Involved in Its Forward Computation

Notes:

<!-- Slide number: 66 -->

Types of Recurrent Neural Networks


Andrej Karpathy, 2015, The Unreasonable Effectiveness of Recurrent Neural Networks

Notes:

<!-- Slide number: 67 -->

Backpropagation Through Time (BPTT)

Notes:

<!-- Slide number: 68 -->

Recurrent Neural Network Problem

Notes:

<!-- Slide number: 69 -->

Long Short-term Memory Network



Colah, 2015, Understanding LSTMs Networks

Notes:

Long short-term memory (LSTM) applies to the scenario with a large gap between the relevant information and the point where it is needed. It can connect previous information for long periods of time to the present task.

<!-- Slide number: 70 -->

Gated Recurrent Unit (GRU)


Notes:

As a variant of LSTM, GRU combines the Forget Gate and the Input Gate into a single Update Gate. It mixes the cell state and hidden state and also incorporates other changes. As a popular variant, its final model is simpler than the standard LSTM model. [] indicates concatenation, * indicates element product, and· indicates matrix multiplication.

<!-- Slide number: 71 -->

Generative Adversarial Network (GAN)

Generative Adversarial Network is a framework that trains generator G and discriminator D through the adversarial process. Through the adversarial process, the discriminator can tell whether the sample from the generator is fake or real. GAN adopts a mature BP algorithm.

(1) Generator G: The input is noise z, which complies with manually selected prior probability distribution, such as even distribution and Gaussian distribution. The generator adopts the network structure of the multilayer perceptron (MLP), uses maximum likelihood estimation (MLE) parameters to represent the derivable mapping G(z), and maps the input space to the sample space.

(2) Discriminator D: The input is the real sample x and the fake sample G(z), which are tagged as real and fake respectively. The network of the discriminator can use the MLP carrying parameters. The output is the probability D(G(z)) that determines whether the sample is a real or fake sample.

GAN can be applied to scenarios such as image generation, text generation, speech enhancement, image super-resolution.

Notes:

GAN: Traditional deep generative models require Markov chains or maximum likelihood estimation (MLE), which brings a lot of probability issues difficult to compute.

<!-- Slide number: 72 -->

GAN Architecture

Generator/Discriminator


Notes:

There are two kinds of models in a GAN: generative model and discriminative model. The discriminative model is used to predict the classification based on input variables. The generative model is used to randomly generate observable-data values, typically using some given hidden parameters. Example:

Generative model

Generates a new cat image that does not exist in the dataset with random data given.

Discriminative model

Discriminates the animal (whether it is a cat or not) in a given image.

<!-- Slide number: 73 -->

Generative Model and Discriminative Model

Generative network

Generates sample data

Input: Gaussian white noise vector z

Output: sample data vector x

D

y

x

G

z

x

Notes:

<!-- Slide number: 74 -->

Training Rules of GAN

Optimization objective:

Value function

In the early training stage, when the outcome of G is very poor, D determines that the generated sample is fake with high confidence, because the sample is obviously different from training data. In this case, log(1-D(G(z))) is saturated (where the gradient is 0, and iteration cannot be performed). Therefore, we choose to train G only by minimizing [-log(D(G(z))].

Notes:

When the value function is used during training, the GAN is called minimax GAN. When [-log(D(G(z))] is used during training, the GAN is called non-saturating GAN.

<!-- Slide number: 75 -->

Deep Learning Summary

Training Rules

Activation Function

Normalizer

Optimizer

Types of Neural Networks

Common Problems

Notes:

<!-- Slide number: 76 -->

Data Imbalance (1)

Problem description: In the dataset consisting of various task categories, the number of samples varies greatly from one category to another. One or more categories in the predicted categories contain very few samples.

For example, in an image recognition experiment, more than 2,000 categories among a total of 4251 training images contain just one image each. Some of the others have 2-5 images.

Impacts:

Due to the unbalanced number of samples, we cannot get the optimal real-time result because model/algorithm never examines categories with very few samples adequately.

Since few observation objects may not be representative for a class, we may fail to obtain adequate samples for verification and test.

Notes:

<!-- Slide number: 77 -->

Data Imbalance (2)

Notes:

Random undersampling: Classes with sufficient observations are randomly deleted to enable significant relative ratio between the two classes. Although this method is easy to use, the deleted data packet may contain important information about the prediction class.

Advantage

It increases the running time. In addition, when the training dataset is large, a storage problem may be resolved by reducing samples.

Disadvantage

It discards potentially valuable information that is important for building a rule classifier.

Samples selected by random undersampling may have a bias. It does not accurately represent the majority. Therefore, the result for an actual test dataset is not accurate.

Random oversampling: For unbalanced classes, we randomly increase the number of observations by copying existing samples. Ideally, we have sufficient samples, but oversampling may lead to overfitting training data.

Advantage

It does not cause information loss.

It outperforms undersampling.

Disadvantage

It increases the likelihood of overfitting because of the replication of minority class event.

<!-- Slide number: 78 -->

Data Imbalance (2)

Notes:

Synthetic Minority Oversampling Technique (SMOTE) — This technique requires us to obtain observations of the unbalanced classes using a synthetic method, which is similar to the existing k-NN method. The question is what we can do when the number of ob...