TensorFlow 2.x Basics

Common Modules of TensorFlow 2.x

Basic Steps of Deep Learning Development

Notes:

<!-- Slide number: 11 -->

TensorFlow

TensorFlow is Google's second-generation open-source software library for digital computing. The TensorFlow computing framework supports various deep learning algorithms and multiple computing platforms, ensuring high system stability.

! (Picture4.jpg)

Image source: https://www.TensorFlow.org/

Notes:

<!-- Slide number: 12 -->

Features of TensorFlow

Scalability

Multi-lingual


GPU

Multi-platform

Powerful computing

Distributed

Notes:

Multi-platform: All platforms that support the Python development environment support TensorFlow. However, TensorFlow depends on other software such as the NVIDIA CUDA Toolkit and cuDNN to access a supported GPU.

GPU: TensorFlow supports certain NVIDIA GPUs, which are compatible with NVIDIA CUDA Toolkit versions that meet specific performance standards.

Distributed: TensorFlow supports distributed computing, allowing graphics to be computed on different processes. These processes may be located on different servers.

Multi-lingual: TensorFlow primarily uses Python for programming. It also supports C++, Java, and Go application programming interfaces (APIs), but cannot guarantee stability when these languages are used. Similarly, TensorFlow cannot ensure stability when third parties use TensorFlow based on C#, Haskell, Julia, Rust, Ruby, Scala, and R (even PHP). Google recently released a mobile-optimized TensorFlow Lite library for running TensorFlow applications on Android.

Powerful computing: TensorFlow can achieve optimal performance on Google TPUs, but still strives to deliver high performance on various platforms, which include not only servers and desktops, but also embedded systems and mobile devices.

Scalability: One of the main advantages of TensorFlow lies in its modular, scalable, and flexible design. Developers can easily port models among the CPU, GPU, and TPU with a few code changes. Python developers can develop their own models by using native and low-level APIs (or core APIs) of TensorFlow, or develop built-in models by using advanced API libraries of TensorFlow. TensorFlow has many built-in and distributed libraries. TensorFlow can be overlaid with an advanced deep learning framework (such as Keras) to serve as an advanced API.

<!-- Slide number: 13 -->

TensorFlow - Distributed

TensorFlow can run on different computers:

From smartphones to computer clusters, to generate desired training models.

Currently, supported native distributed deep learning frameworks include only TensorFlow, CNTK, Deeplearning4J, and MXNet.

When a single GPU is used, most deep learning frameworks rely on cuDNN, and therefore support almost the same training speed, provided that the hardware computing capabilities or allocated memories slightly differ. However, for large-scale deep learning, massive data makes it difficult for the single GPU to complete training in a limited time. To handle such cases, TensorFlow enables distributed training.

Notes:

<!-- Slide number: 14 -->

Why TensorFlow?

TensorFlow is considered as one of the best libraries for neural networks, and can reduce difficulty in deep learning development. In addition, as it is open-source, it can be conveniently maintained and updated, thus the efficiency of development can be improved.

Keras, ranking third in the number of stars on GitHub, is packaged into an advanced API of TensorFlow 2.0, which makes TensorFlow 2.x more flexible, and easier to debug.


Demand on the recruitment market

Notes:

<!-- Slide number: 15 -->

TensorFlow 2.x vs. TensorFlow 1.x

Disadvantages of TensorFlow 1.0:

After a tensor is created in TensorFlow 1.0, the result cannot be returned directly. To obtain the result, the session mechanism needs to be created, which includes the concept of graph, and code cannot run without session.run. This style is more like the hardware programming language VHDL.

Compared with some simple frameworks such as PyTorch, TensorFlow 1.0 adds the session and graph concepts, which are inconvenient for users.

It is complex to debug TensorFlow 1.0, and its APIs are disordered, making it difficult for beginners. Learners will come across many difficulties in using TensorFlow 1.0 even after gaining the basic knowledge. As a result, many researchers have turned to PyTorch.

Notes:

<!-- Slide number: 16 -->

TensorFlow 2.x vs. TensorFlow 1.x

Features of TensorFlow 2.x:

Advanced API Keras:

Easy to use: The graph and session mechanisms are removed. What you see is what you get, just like Python and PyTorch.

Major improvements:

The core function of TensorFlow 2.x is the dynamic graph mechanism called eager execution. It allows users to compile and debug models like normal programs, making TensorFlow easier to learn and use.

Multiple platforms and languages are supported, and compatibility between components can be improved via standardization on exchange formats and alignment of APIs.

Deprecated APIs are deleted and duplicate APIs are reduced to avoid confusion.

Compatibility and continuity: TensorFlow 2.x provides a module enabling compatibility with TensorFlow 1.x.

The tf.contrib module is removed. Maintained modules are moved to separate repositories. Unused and unmaintained modules are removed.

Notes:

<!-- Slide number: 17 -->

Mainstream Development Frameworks

TensorFlow 2.x Basics

Common Modules of TensorFlow 2.x

Basic Steps of Deep Learning Development

Notes:

<!-- Slide number: 18 -->

Tensors

Tensors are the most basic data structures in TensorFlow. All data is encapsulated in tensors.

Tensor: a multidimensional array

A scalar is a rank-0 tensor. A vector is a rank-1 tensor. A matrix is a rank-2 tensor.

In TensorFlow, tensors are classified into:

Constant tensors

Variable tensors


One-dimensional tensor

Two-dimensional tensor

Three-dimensional tensor


Four-dimensional tensor

Five-dimensional tensor

Six-dimensional tensor

Notes:

Jupyter; tensor introduction chapter

<!-- Slide number: 19 -->

Basic Operations of TensorFlow 2.x

The following describes common APIs in TensorFlow by focusing on code. The main content is as follows:

Methods for creating constants and variables

Tensor slicing and indexing

Dimension changes of tensors

Arithmetic operations on tensors

Tensor concatenation and splitting

Tensor sorting

Notes:

The following describes common APIs in TensorFlow by focusing on code.

<!-- Slide number: 20 -->

Eager Execution Mode of TensorFlow 2.x

Static graph: TensorFlow 1.x using static graphs (graph mode) separates computation definition and execution by using computational graphs. This is a declarative programming model. In graph mode, developers need to build a computational graph, start a session, and then input data to obtain an execution result.

Static graphs are advantageous in distributed training, performance optimization, and deployment, but inconvenient for debugging. Executing a static graph is similar to invoking a compiled C language program, and internal debugging cannot be performed in this case. Therefore, eager execution based on dynamic computational graphs emerges.

Eager execution is a command-based programming method, which is the same as native Python. A result is returned immediately after an operation is performed.

Notes:

<!-- Slide number: 21 -->

AutoGraph

Eager execution is enabled in TensorFlow 2.x by default. Eager execution is intuitive and flexible for users (easier and faster to run a one-time operation), but may compromise performance and deployability.

To achieve optimal performance and make a model deployable anywhere, you can run @tf.function to add a decorator to build a graph from a program, making Python code more efficient.

tf.function can build a TensorFlow operation in the function into a graph. In this way, this function can be executed in graph mode. Such practice can be considered as encapsulating the function as a TensorFlow operation of a graph.

Notes:

<!-- Slide number: 22 -->

Mainstream Development Frameworks

TensorFlow 2.x Basics

Common Modules of TensorFlow 2.x

Basic Steps of Deep Learning Development

Notes:

<!-- Slide number: 23 -->

Common Modules of TensorFlow 2.x (1)

tf: Functions in the tf module are used to perform common arithmetic operations, such as tf.abs (calculating an absolute value), tf.add (adding elements one by one), and tf.concat (concatenating tensors). Most operations in this module can be performed by NumPy.

tf.errors: error type module of TensorFlow

tf.data: implements operations on datasets.

Input pipes created by tf.data are used to read training data. In addition, data can be easily input from memories (such as NumPy).

tf.distributions: implements various statistical distributions.

The functions in this module are used to implement various statistical distributions, such as Bernoulli distribution, uniform distribution, and Gaussian distribution.

Notes:

<!-- Slide number: 24 -->

Common Modules of TensorFlow 2.x (2)

tf.io.gfile: implements operations on files.

Functions in this module can be used to perform file I/O operations, copy files, and rename files.

tf.image: implements operations on images.

Functions in this module include image processing functions. This module is similar to OpenCV, and provides functions related to image luminance, saturation, phase inversion, cropping, resizing, image format conversion (RGB to HSV, YUV, YIQ, or gray), rotation, and sobel edge detection. This module is equivalent to a small image processing package of OpenCV.

tf.keras: a Python API for invoking Keras tools.

This is a large module that enables various network operations.

Notes:

<!-- Slide number: 25 -->

Keras Interface

TensorFlow 2.x recommends Keras for network building. Common neural networks are included in Keras.layers.

Keras is a high-level API used to build and train deep learning models. It can be used for rapid prototype design, advanced research, and production. It has the following three advantages:

Easy to useKeras provides simple and consistent GUIs optimized for common cases. It provides practical and clear feedback on user errors.

Modular and composableYou can build Keras models by connecting configurable building blocks together, with little restriction.

Easy to extendYou can customize building blocks to express new research ideas, create layers and loss functions, and develop advanced models.

Notes:

<!-- Slide number: 26 -->

Common Keras Methods and Interfaces

The following describes common methods and interfaces of tf.keras by focusing on code. The main content is as follows:

Dataset processing: datasets and preprocessing

Neural network model creation: Sequential, Model, Layers...

Network compilation: compile, Losses, Metrics, and Optimizers

Network training and evaluation: fit, fit_generator, and evaluate

Notes:

<!-- Slide number: 27 -->

Mainstream Development Frameworks

TensorFlow 2.x Basics

Common Modules of TensorFlow 2.x

Basic Steps of Deep Learning Development

Notes:

<!-- Slide number: 28 -->

TensorFlow Environment Setup in Windows 10

Environment setup in Windows 10:

Operating system: Windows 10

pip software built in Anaconda 3 (adapting to Python 3)

TensorFlow installation:

Open Anaconda Prompt and run the pip command to install TensorFlow.

Run pip install TensorFlow in the command line interface.


Notes:

<!-- Slide number: 29 -->

TensorFlow Environment Setup in Ubuntu/Linux

The simplest way for installing TensorFlow in Linux is to run the pip command.

pip command: pip install TensorFlow==2.1.0


Notes:

If the installation is slow, replace the existing mirror with a Tsinghua mirror, and run the following command on the terminal:

pip install pip U

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

<!-- Slide number: 30 -->

TensorFlow Development Process

Data preparation

Data exploration

Data processing

Network construction

Defining a network structure.

Defining loss functions, selecting optimizers, and defining model evaluation indicators.

Model training and verification

Model saving

Model restoration and invoking

Data preparation

Model deployment and application

Model verification

Model training

Model definition

Notes:

<!-- Slide number: 31 -->

Project Description

Handwritten digit recognition is a common image recognition task where computers recognize text in handwriting images. Different from printed fonts, handwriting of different people has different sizes and styles, making it difficult for computers to recognize handwriting. This project applies deep learning and TensorFlow tools to train and build models based on the MNIST handwriting dataset.

7

1

0

5


Handwritten digit recognition

Notes:

<!-- Slide number: 32 -->

Data Preparation

MNIST datasets

Download the MNIST datasets from http://yann.lecun.com/exdb/mnist/.

The MNIST datasets consist of a training set and a test set.

Training set: 60,000 handwriting images and corresponding labels

Test set: 10,000 handwriting images and corresponding labels

Examples


Corresponding labels

[0,0,0,0,0,1,0,0,0,0]

[0,0,0,0,0,0,0,0,0,1]

[0,0,0,0,0,0,0,1,0,0]

[0,0,0,1,0,0,0,0,0,0]

[0,0,0,0,1,0,0,0,0,0]

Notes:

<!-- Slide number: 33 -->

Network Structure Definition (1)

Softmax regression model

The softmax function is also called normalized exponential function. It is a derivative of the binary classification function sigmoid in terms of multi-class classification. The following figure shows the calculation method of softmax.


Notes:

<!-- Slide number: 34 -->

Network Structure Definition (2)

The process of model establishment is the core process of network structure definition.

The network operation process defines how model output is calculated based on input.

Matrix multiplication and vector addition are used to express the calculation process of softmax.



Notes:

<!-- Slide number: 35 -->

Network Structure Definition (3)

TensorFlow-based softmax regression model

import tensorflow

import tensorflow as tf

##define input variables with operator symbol variables.

we use a variable to feed data into the graph through the placeholders X. Each input image is flattened into a 784-dimensional vector. In this case, the shape of the tensor is , None indicates can be of any length.

X = tf.placeholder(tf.float32, )

The variable that can be modified is used to indicate the weight w and bias b. The initial values are set to 0.

w = tf.Variable(tf.zeros([784,10]))

b = tf.Variable(tf.zeros([10]))

If tf.matmul(x, w) is used to indicate that x is multiplied by w, the Soft regression equation is y = softmax(wx+b)'

y = tf.nn.softmax(tf.matmul(x,w)+b)

Notes:

<!-- Slide number: 36 -->

Network Compilation

Model compilation involves the following two parts:

Loss function selection

In machine learning/deep learning, an indicator needs to be defined to indicate whether a model is proper. This indicator is called cost or loss, and is minimized as far as possible. In this project, the cross entropy loss function is used.

Gradient descent method

A loss function is constructed for an original model needs to be optimized by using an optimization algorithm, to find optimal parameters and further minimize a value of the loss function. Among optimization algorithms for solving machine learning parameters, the gradient descent-based optimization algorithm (Gradient Descent) is usually used.

model.compile(optimizer=tf.train.AdamOptimizer(),

loss=tf.keras.losses.categorical_crossentropy,

metrics= )

Notes:

<!-- Slide number: 37 -->

Model Training

Training process:

All training data is trained through batch iteration or full iteration. In the experiment, all data is trained five times.

In TensorFlow, model.fit is used for training, where epoch indicates the number of training iterations.


Notes:

<!-- Slide number: 38 -->

Model Evaluation

You can test the model using the test set, compare predicted results with actual ones, and find correctly predicted labels, to calculate the accuracy of the test set.


Loss value

Accuracy

Notes:

<!-- Slide number: 39 -->

In TensorFlow 2.x, eager execution is enabled by default. ( )

True

False

Which of the following statements about tf.keras.Model and tf.keras.Sequential is incorrect when the tf.keras interface is used to build a network model? ( )

tf.keras.Model supports network models with multiple inputs, while tf.keras.Sequential does not.

tf.keras.Model supports network models with multiple outputs, while tf.keras.Sequential does not.

tf.keras.Model is recommended for model building when a sharing layer exists on the network.

tf.keras.Sequential is recommended for model building when a sharing layer exists on the network.

Notes:

Answers:

A

D

<!-- Slide number: 40 -->

This chapter describes the following content by focusing on code: Features of common deep learning frameworks, including PyTorch and TensorFlowBasic syntax and common modules of TensorFlow 2.x Development procedure of TensorFlow.

Notes:

<!-- Slide number: 41 -->

Official TensorFlow website: https://tensorflow.google.cn

Official PyTorch website: https://PyTorch.org/

Notes:

<!-- Slide number: 42 -->

Notes: