<!-- Slide number: 1 -->
Shu Xiaodong/swx640908
2020.02.16
New
Notes:
<!-- Slide number: 2 -->
Atlas AI Computing Platform
Notes:
<!-- Slide number: 3 -->
This chapter describes Huawei's Ascend AI chips, and hardware and software architectures of Ascend chips, and full-stack all-scenario solutions of Ascend AI chips.
Notes:
<!-- Slide number: 4 -->
On completion of the course, you will be able to:
Get an overview of AI chips.
Understand hardware and software architectures of Huawei Ascend chips.
Learn about Huawei Atlas AI computing platform.
Understand industry applications of Atlas.
Notes:
<!-- Slide number: 5 -->
Overview of AI Chips
Summary
Classification of AI Chips
Current Status of AI Chips
GPU, TPU, and Ascend 310 Design Comparison
Ascend AI Processors
Hardware Architecture of Ascend Chips
…
Notes:
<!-- Slide number: 6 -->
This section is an overview of AI chips, including the introduction, classification, and status of AI chips, comparison between GPUs and CPUs, and introduction of Ascend AI processors.
Notes:
<!-- Slide number: 7 -->
Definition
Four elements of AI: data, algorithm, scenario, and computing power
AI chips, also known as AI accelerators, are function modules that process massive computing tasks in AI applications.
Notes:
<!-- Slide number: 8 -->
Overview of AI Chips
Summary
Classification of AI Chips
Current Status of AI Chips
Design Comparison of GPUs and CPUs
Ascend AI Processors
Hardware Architecture of Ascend Chips
…
Notes:
<!-- Slide number: 9 -->
Classification of AI Chips
AI Chips can be divided into four types by technical architecture:
A central processing unit (CPU): a super-large-scale integrated circuit, which is the computing core and control unit of a computer. It can interpret computer instructions and process computer software data.
A graphics processing unit (GPU): a display core, visual processor, and display chip. It is a microprocessor that processes images on personal computers, workstations, game consoles, and mobile devices, such as tablet computers and smart phones.
An application specific integrated circuit (ASIC): an integrated circuit designed for a specific purpose.
A field programmable gate array (FPGA): designed to implement functions of a semi-customized chip. The hardware structure can be flexibly configured and changed in real time based on requirements.
Notes:
<!-- Slide number: 10 -->
Overview of AI Chips
Summary
Classification of AI Chips
Current Status of AI Chips
Design Comparison of GPUs and CPUs
Ascend AI Processors
Hardware Architecture of Ascend Chips
…
Notes:
<!-- Slide number: 11 -->
Current Status of AI Chips - CPU
Central processing unit (CPU)
The computer performance has been steadily improved based on the Moore's Law.
The CPU cores added for performance enhancement also increase power consumption and cost.
Extra instructions have been introduced and the architecture has been modified to improve AI performance.
Instructions, such as AVX512, have been introduced into Intel processors (CISC architecture) and vector computing modules, such as FMA, into the ALU computing module.
Instruction sets including Cortex A have been introduced into ARM (RISC architecture), which will be upgraded continuously.
Despite that boosting the processor frequency can elevate the performance, the high frequency will cause huge power consumption and overheating of the chip as the frequency reaches the ceiling.
Notes:
In earlier years, the computer performance had been steadily improved based on the Moore's Law to meet the needs of users. People impose increasingly high requirements on computer performance, while performance improvement mostly depends on advancement of underlying hardware, which accelerates upper-layer application software. In recent years, improvement brought by the Moore's Law has slowed down. Hardware development gradually encounters physical bottlenecks. Limits on heat dissipation and power consumption make it difficult to further improve the performance of serial programs in the traditional CPU architecture. The current situation drives the industry to constantly look for an architecture and the corresponding software framework more suitable to the post-Moore's Law era.
The multi-core processor is developed to better meet the hardware speed requirements of software. Intel Core i7 series processors use the parallel instruction processor cores constructed by four independent kernels based on the x86 instruction set, which accelerates the processor programs to some extent. However, the number of kernels cannot increase infinitely, and most traditional CPU programs are written by serial programming. Therefore, a large number of programs cannot be accelerated.
<!-- Slide number: 12 -->
Current Status of AI Chips - GPU
Graph processing unit (GPU)
GPU performs remarkably in matrix computing and parallel computing and plays a key role in heterogeneous computing. It was first introduced to the AI field as an acceleration chip for deep learning. Currently, the GPU ecosystem has matured.
Using the GPU architecture, NVIDIA focuses on the following two aspects of deep learning:
Diversifying the ecosystem: It has launched the cuDNN optimization library for neural networks to improve usability and optimize the GPU underlying architecture.
Improving customization: It supports various data types, including int8 in addition to float32; introduces modules dedicated for deep learning. For example, the optimized architecture of Tensor cores has been introduced, such as the TensorCore of V100.
The existing problems include high costs and latency and low energy efficiency.
Notes:
The CPU focuses on logic control in instruction execution, while the GPU has outstanding advantages in large-scale, intensive, and parallel data computing. Program optimization generally requires collaboration of the CPU and GPU.
<!-- Slide number: 13 -->
Current Status of AI Chips - TPU
Tensor processing unit (TPU)
Since 2006, Google has sought to apply the design concept of ASICs to the neural network field and released TPU, a customized AI chip that supports TensorFlow, which is an open-source deep learning framework.
Massive systolic arrays and large-capacity on-chip storage are adopted to accelerate the most common convolution operations in deep neural networks.
Systolic arrays optimize matrix multiplication and convolution operations to elevate computing power and lower energy consumption.
Notes:
The CPU focuses on logic control in instruction execution, while the GPU has outstanding advantages in large-scale, intensive, and parallel data computing. Program optimization requires collaboration of the CPU and GPU.
<!-- Slide number: 14 -->
Current Status of AI Chips - FPGA
Field programmable gate array (FPGA)
Using the HDL programmable mode, FPGAs are highly flexible, reconfigurable and re-programmable, and customizable.
Multiple FPGAs can be used to load the DNN model on the chips to lower computing latency. FPGAs outperform GPUs in terms of computing performance. However, the optimal performance cannot be achieved due to continuous erasing and programming. Besides, redundant transistors and cables, logic circuits with the same functions occupy a larger chip area.
The reconfigurable structure lowers supply and R&D risks. The cost is relatively flexible depending on the purchase quantity.
The design and tapeout processes are decoupled. The development period is long, generally half a year. The entry barrier is high.
Notes:
<!-- Slide number: 15 -->
Overview of AI Chips
Summary
Classification of AI Chips
Current Status of AI Chips
Design Comparison of GPUs and CPUs
Ascend AI Processors
Hardware Architecture of Ascend Chips
Software Architecture of Ascend Chips
…
Notes:
<!-- Slide number: 16 -->
Design Comparison of GPUs and CPUs
GPUs are designed for massive data of the same type independent from each other and pure computing environments that do not need to be interrupted.
Each GPU comprises several large-sized parallel computing architectures with thousands of smaller cores designed to handle multiple tasks simultaneously.
Throughput-oriented design
With many ALUs and few caches, which improve services for threads, unlike those in CPU. The cache merges access to DRAM, causing latency.
The control unit performs combined access.
A large number of ALUs process numerous threads concurrently to cover up the latency.
Specialized in computing-intensive and easy-to-parallel programs
Notes:
<!-- Slide number: 17 -->
Design Comparison of GPUs and CPUs
CPUs need to process different data types in a universal manner, perform logic judgment, and introduce massive branch jumps and interrupted processing.
Composed of several cores optimized for sequential serial processing
Low-latency design
The powerful ALU unit can complete the calculation in a short clock cycle.
Publicité
The large cache lowers latency.
High clock frequency
Complex logic control unit, multi-branch programs can reduce latency through branch prediction.
For instructions that depend on the previous instruction result, the logic unit determines the location of the instructions in the pipeline to speed up data forwarding.
Specialized in logic control and serial operation
ALU
ALU
Control
ALU
ALU
Cache
DRAM
Notes:
<!-- Slide number: 18 -->
Overview of AI Chips
Summary
Classification of AI Chips
Current Status of AI Chips
Design Comparison of GPUs and CPUs
Ascend AI Processors
Hardware Architecture of Ascend Chips
Software Architecture of Ascend Chips
…
Notes:
<!-- Slide number: 19 -->
Ascend AI Processors
Neural-network processing unit (NPU): uses a deep learning instruction set to process a large number of human neurons and synapses simulated at the circuit layer. One instruction is used to process a group of neurons.
Typical NPUs: Huawei Ascend AI chips, Cambricon chips, and IBM TrueNorth
Ascend-Mini
Architecture: Da Vinci
Half precision (FP16): 8 Tera-FLOPS
Integer precision (INT8): 16 Tera-OPS
16-channel full-HD video decoder: H.264/H.265
1-channel full-HD video decoder: H.264/H.265
Max. power: 8W
12nm FFC
Ascend-Max
Architecture: Da Vinci
Half precision (FP16): 256 Tera-FLOPS
Integer precision (INT8): 512 Tera-OPS
128-channel full-HD video decoder: H.264/H.265
Max. power: 350W
7nm
Notes:
<!-- Slide number: 20 -->
This section describes AI chips, including classification of AI chips by technologies and functions, AI chip ecosystem, and comparison between GPUs and CPUs.
Notes:
<!-- Slide number: 21 -->
Overview of AI Chips
Hardware Architecture of Ascend Chips
Logic Architecture of Ascend AI Processors
Da Vinci Architecture
Software Architecture of Ascend Chips
Huawei Atlas AI Computing Platform
Industry Applications of Atlas
Notes:
<!-- Slide number: 22 -->
This section describes the hardware architecture of Ascend chips, including the logic architecture of the Ascend AI processors and Da Vinci architecture.
Notes:
<!-- Slide number: 23 -->
Logic Architecture of Ascend AI Processors
Ascend AI processor consist of:
Control CPU
AI computing engine, including AI core and AI CPU
Multi-layer system-on-chip (SoC) caches or buffers
Digital vision pre-processing (DVPP) module
Notes:
<!-- Slide number: 24 -->
Logic Architecture of Ascend AI Processors
Network
Card
USB
Interface
PCIe
Interface
Ascend AI Chip
Task
Scheduler
Control CPU
AI Core
AI CPU
Bus
Main Memory
L2 Buffer
DVPP
GPI0/I2C .etc
DDR/HBM Interface
Notes:
<!-- Slide number: 25 -->
Overview of AI Chips
Hardware Architecture of Ascend Chips
Logic Architecture of Ascend AI Processors
Da Vinci Architecture
Software Architecture of Ascend Chips
Huawei Atlas AI Computing Platform
Industry Applications of Atlas
Notes:
<!-- Slide number: 26 -->
Ascend AI Computing Engine - Da Vinci Architecture
One of the four major architectures of Ascend AI processors is the AI computing engine, which consists of the AI core (Da Vinci architecture) and AI CPU. The Da Vinci architecture developed to improve the AI computing power serves as the core of the Ascend AI computing engine and AI processor.
Notes:
<!-- Slide number: 27 -->
Da Vinci Architecture (AI Core)
Main components of the Da Vinci architecture:
Computing unit: It consists of the cube unit, vector unit, and scalar unit.
Storage system: It consists of the on-chip storage unit of the AI core and data channels.
Control unit provides instruction control for the entire computing process. It is equivalent to the command center of the AI core and is responsible for the running of the entire AI core.
Notes:
<!-- Slide number: 28 -->
Da Vinci Architecture (AI Core)
Matrix
Calculation Unit
ImgCol
Accumulator
Unzip
Input Buffer
Transpose
Storage Conversion unit
ReLU
Publicité
FP16<->FP32
Input Buffer Controller
Vector Calculation Unit
Output Buffer
General
Register
Scalar Computing Unit
Special
Register
Scalar Instruction Processing Queue
Matrix Operation Queue
Configure Port
Event Synchroniza-tion Module
Command Transmission Module
System Control Module
Bus Interface Unit
Vector Operation Queue
Instruction Cache
Storage Operation Queue
Bus(L2 Buffer/HBM/DDR)
Notes:
<!-- Slide number: 29 -->
Da Vinci Architecture (AI Core) - Computing Unit
Three types of basic computing units: cube, vector, and scalar units, which correspond to matrix, vector and scalar computing modes respectively.
Cube computing unit: The matrix computing unit and accumulator are used to perform matrix-related operations. Completes a matrix (4096) of 16x16 multiplied by 16x16 for FP16, or a matrix (8192) of 16x32 multiplied by 32x16 for the INT8 input in a shot.
Vector computing unit: Implements computing between vectors and scalars or between vectors. This function covers various basic computing types and many customized computing types, including computing of data types such as FP16, FP32, INT32, and INT8.
Scalar computing unit: Equivalent to a micro CPU, the scalar unit controls the running of the entire AI core. It implements loop control and branch judgment for the entire program, and provides the computing of data addresses and related parameters for cubes or vectors as well as basic arithmetic operations.
Notes:
<!-- Slide number: 30 -->
Da Vinci Architecture (AI Core) - Computing Unit
Matrix
Calculation Unit
ImgCol
Accumulator
Unzip
Input Buffer
Transpose
Storage Conversion unit
ReLU
FP16<->FP32
Input Buffer Controller
Vector Calculation Unit
Output Buffer
Scalar Computing Unit
Special
Register
Scalar Instruction Processing Queue
Matrix Operation Queue
Configure Port
Event Synchroniza-tion Module
Command Transmission Module
System Control Module
Bus Interface Unit
Vector Operation Queue
Instruction Cache
Storage Operation Queue
Bus(L2 Buffer/HBM/DDR)
General
Register
Notes:
<!-- Slide number: 31 -->
Da Vinci Architecture (AI Core) - Storage System (1)
The storage system of the AI core is composed of the storage unit and corresponding data channel.
The storage unit consists of the storage control unit, buffer, and registers:
Storage control unit: The cache at a lower level than the AI core can be directly accessed through the bus interface. The memory can also be directly accessed through the DDR or HBM. A storage conversion unit is set as a transmission controller of the internal data channel of the AI core to implement read/write management of internal data of the AI core between different buffers. It also completes a series of format conversion operations, such as zero padding, Img2Col, transposing, and decompression.
Input buffer: The buffer temporarily stores the data that needs to be frequently used so the data does not need to be read from the AI core through the bus interface each time. This mode reduces the frequency of data access on the bus and the risk of bus congestion, thereby reducing power consumption and improving performance.
Output buffer: The buffer stores the intermediate results of computing at each layer in the neural network, so that the data can be easily obtained for next-layer computing. Reading data through the bus involves low bandwidth and long latency, whereas using the output buffer greatly improves the computing efficiency.
Register: Various registers in the AI core are mainly used by the scalar unit.
Notes:
<!-- Slide number: 32 -->
Da Vinci Architecture (AI Core) - Storage System (2)
Data channel: path for data flowing in the AI core during execution of computing tasks
A data channel of the Da Vinci architecture is characterized by multiple-input single-output. Considering various types and a large quantity of input data in the computing process on the neural network, parallel inputs can improve data inflow efficiency. On the contrary, only an output feature matrix is generated after multiple types of input data are processed. The data channel with a single output of data reduces the use of chip hardware resources.
Notes:
<!-- Slide number: 33 -->
Da Vinci Architecture (AI Core) - Storage System (2)
Matrix
Calculation Unit
ImgCol
Accumulator
Transpose
Input Buffer
Unzip
Storage Conversion unit
ReLU
FP16<->FP32
Input Buffer Controller
Vector Calculation Unit
Output Buffer
General
Register
Scalar Computing Unit
Special
Register
Scalar Instruction Processing Queue
Matrix Operation Queue
Configure Port
Event Synchroniza-tion Module
Command Transmission Module
System Control Module
Bus Interface Unit
Vector Operation Queue
Instruction Cache
Storage Operation Queue
Bus(L2 Buffer/HBM/DDR)
Notes:
<!-- Slide number: 34 -->
Da Vinci Architecture (AI Core) - Control Unit (1)
The control unit consists of the system control module, instruction cache, scalar instruction processing queue, instruction transmitting module, matrix operation queue, vector operation queue, storage conversion queue, and event synchronization module.
System control module: Controls the execution process of a task block (minimum task computing granularity for the AI core). After the task block is executed, the system control module processes the interruption and reports the status. If an error occurs during the execution, the error status is reported to the task scheduler.
Instruction cache: Prefetches subsequent instructions in advance during instruction execution and reads multiple instructions into the cache at a time, improving instruction execution efficiency.
Scalar instruction procession queue: After being decoded, the instructions are imported into a scalar queue to implement address decoding and operation control. The instructions include matrix computing instructions, vector calculation instructions, and storage conversion instructions.
Instruction transmitting module: Reads the configured instruction addresses and decoded parameters in the scalar instruction queue, and sends them to the corresponding instruction execution queue according to the instruction type. The scalar instructions reside in the scalar instruction processing queue for subsequent execution.
Notes:
<!-- Slide number: 35 -->
Da Vinci Architecture (AI Core) - Control Unit (2)
Instruction execution queue: Includes a matrix operation queue, vector operation queue, and storage conversion queue. Different instructions enter corresponding operation queues, and instructions in the queues are executed according to the entry sequence.
Event synchronization module: Controls the execution status of each instruction pipeline in real time, and analyzes dependence relationships between different pipelines to resolve problems of data dependence and synchronization between instruction pipelines.
Notes:
<!-- Slide number: 36 -->
Da Vinci Architecture (AI Core) - Control Unit (2)
Matrix
Calculation Unit
ImgCol
Accumulator
Transpose
Input Buffer
Unzip
Storage Conversion unit
ReLU
Publicité
FP16<->FP32
Input Buffer Controller
Vector Calculation Unit
Output Buffer
General
Register
Scalar Computing Unit
Special
Register
Matrix Operation Queue
Scalar Instruction Processing Queue
Configure Port
Event Synchroniza-tion Module
System Control Module
Command Transmission Module
Bus Interface Unit
Vector Operation Queue
Instruction Cache
Storage Operation Queue
Bus(L2 Buffer/HBM/DDR)
Notes:
<!-- Slide number: 37 -->
This section describes the hardware architecture of Ascend chips, including the computing unit, storage unit, and control unit of the core Da Vinci architecture.
Notes:
<!-- Slide number: 38 -->
Overview of AI Chips
Hardware Architecture of Ascend Chips
Software Architecture of Ascend Chips
Logic Architecture of Ascend 310
Neural Network Software Flow of Ascend 310
Huawei Atlas AI Computing Platform
Industry Applications of Atlas
Notes:
<!-- Slide number: 39 -->
This section describes the software architecture of Ascend chips, including the logic architecture and neural network software flow of Ascend AI processors.
Notes:
<!-- Slide number: 40 -->
Logic Architecture of Ascend AI Processor Software Stack (1)
L3 application enabling layer: It is an application-level encapsulation layer that provides different processing algorithms for specific application fields. L3 provides various fields with computing and processing engines. It can directly use the framework scheduling capability provided by L2 to generate corresponding NNs and implement specific engine functions.
Generic engine: provides the generic neural network inference capability.
Computer vision engine: encapsulates video or image processing algorithms.
Language and text engine: encapsulates basic processing algorithms for voice and text data.
Notes:
The software stack of the Ascend AI chip consists of four layers and an auxiliary toolchain. The four layers are the application enabling layer (L3), execution framework layer (L2), chip enabling layer (L1), and computing resource layer (L0). The toolchain provides auxiliary capabilities such as program development, compilation and commissioning, application process orchestration, log management, and profiling. The functions of the main components depend on each other in the software stack. They carry data flows, computing flows, and control flows.
<!-- Slide number: 41 -->
Logic Architecture of Ascend AI Processor Software Stack (1)
Tool chain
L3 AI Application
Language Engine
Computer Vision Engine
General Business Execution Engine
Other
Engineering
Management
Compile and Debug
Frame Manager
L2Execution Framework
Process Choreography
AI Model Housekeeper
Offline Model Generator
Process Choreographer
Offline Model Executor
Offline Model Conversion
Comparison Tool
Digital Vision Preprocessing Module
Tensor Acceleration Engine
Run Manager
Drive
L1Chip Enable
Log Management
Task Scheduler
Performance Analysis Tools
L0 Computing Resources
Custom Operator
OS(Linux、Android、EulerOS、LiteOS…)
DVPP Dedicated Hardware
Black Box Tool
AI CPU
AI Core
Notes:
<!-- Slide number: 42 -->
Logic Architecture of Ascend AI Processor Software Stack (2)
L2 execution framework layer: encapsulates the framework calling capability and offline model generation capability. After the application algorithm is developed and encapsulated into an engine at L3, L2 calls the appropriate deep learning framework, such as Caffe or TensorFlow, based on the features of the algorithm to obtain the neural network of the corresponding function, and generates an offline model through the framework manager. After L2 converts the original neural network model into an offline model that can be executed on Ascend AI chips, the offline model executor (OME) transfers the offline model to Layer 1 for task allocation.
L1 chip enabling layer: bridges the offline model to Ascend AI chips. L1 accelerates the offline model for different computing tasks via libraries. Nearest to the bottom-layer computing resources, L1 outputs operator-layer tasks to the hardware.
L0 computing resource layer: provides computing resources and executes specific computing tasks. It is the hardware computing basis of the Ascend AI chip.
Notes:
L2 execution framework layer: encapsulates the framework calling capability and offline model generation capability. After the application algorithm is developed and encapsulated into an engine at L3, L2 calls the appropriate deep learning framework, such as Caffe or TensorFlow, based on the features of the algorithm to obtain the neural network of the corresponding function, and generates an offline model through the framework manager. After L2 converts the original neural network model into an offline model that can be executed on Ascend AI chips, the offline model executor (OME) transfers the offline model to Layer 1 for task allocation.
Online framework: uses a mainstream deep learning open source framework (such as Caffe/TensorFlow). Offline model conversion and loading enable the framework to accelerate computing on the Ascend AI chip.
Offline framework: provides the offline generation and execution capabilities of the neural network, which enables the offline model to have the same capabilities (mainly the inference capability) without the deep learning framework.
Framework manager: includes the offline model generator (OMG), offline model executor (OME), and offline model inference interface; supports model generation, loading, unloading, inference, and computing.
OMG: converts the model files and weight files generated in the Caffe or TensorFlow framework into offline model files, which can be independently executed on the Ascend AI chip.
<!-- Slide number: 43 -->
Logic Architecture of Ascend AI Processor Software Stack (2)
L2 execution framework layer: encapsulates the framework calling capability and offline model generation capability. After the application algorithm is developed and encapsulated into an engine at L3, L2 calls the appropriate deep learning framework, such as Caffe or TensorFlow, based on the features of the algorithm to obtain the neural network of the corresponding function, and generates an offline model through the framework manager. After L2 converts the original neural network model into an offline model that can be executed on Ascend AI chips, the offline model executor (OME) transfers the offline model to Layer 1 for task allocation.
L1 chip enabling layer: bridges the offline model to Ascend AI chips. L1 accelerates the offline model for different computing tasks via libraries. Nearest to the bottom-layer computing resources, L1 outputs operator-layer tasks to the hardware.
L0 computing resource layer: provides computing resources and executes specific computing tasks. It is the hardware computing basis of the Ascend AI chip.
Notes:
OME: loads and unloads offline models, converts successfully loaded model files into instruction sequences that can be executed on the Ascend AI chip, and completes program compilation before execution.
Process orchestrator: provides developers with a development platform for deep learning computing, including computing resources, running framework, and related tools. It is responsible for model generation, loading, and operation scheduling.
L1 chip enabling layer: bridges the offline model to Ascend AI chips. L1 accelerates the offline model for different computing tasks via libraries. Nearest to the bottom-layer computing resources, L1 outputs operator-layer tasks to the hardware.
Tensor boosting engine (TBE): supports accelerated computing of online and offline models by using the standard operator acceleration library and custom operator capabilities. The engine provides operators with complete functions for the L2 layer.
Operation manager: communicates with the L2 layer and provides interfaces of the standard operator acceleration library for the L2 layer to invoke, so that the optimized, executable, and acceleration-available operators can be found for the specific network model to achieve optimal functions.
Task scheduler: processes and distributes computing core functions to the AI CPU or AI core based on the task type, and activates hardware for execution through the driver.
<!-- Slide number: 44 -->
Logic Architecture of Ascend AI Processor Software Stack (2)
L2 execution framework layer: encapsulates the framework calling capability and offline model generation capability. After the application algorithm is developed and encapsulated into an engine at L3, L2 calls the appropriate deep learning framework, such as Caffe or TensorFlow, based on the features of the algorithm to obtain the neural network of the corresponding function, and generates an offline model through the framework manager. After L2 converts the original neural network model into an offline model that can be executed on Ascend AI chips, the offline model executor (OME) transfers the offline model to Layer 1 for task allocation.
L1 chip enabling layer: bridges the offline model to Ascend AI chips. L1 accelerates the offline model for different computing tasks via libraries. Nearest to the bottom-layer computing resources, L1 outputs operator-layer tasks to the hardware.
L0 computing resource layer: provides computing resources and executes specific computing tasks. It is the hardware computing basis of the Ascend AI chip.
Notes:
DVPP module: functions as a multifunctional package oriented to the image and video field. It provides the upper layer with various data (image or video) preprocessing capabilities using dedicated hardware at the bottom layer.
L0 computing resource layer: provides computing resources and executes specific computing tasks. It is the hardware computing basis of the Ascend AI chip.
AI core: functions as the computing core that undertakes computing tasks at large computing power to implement matrix-related computing of the neural network.
AI CPU: provides complex computing and execution control functions to implement generic computing of control operators, scalars, and vectors.
DVPP hardware: preprocesses input data (such as images and video data) and provides data formats that meet computing requirements for the AI core in specific scenarios.
Operating system: collaborates between the preceding three roles to form a complete hardware system, ensuring the execution of the deep neural network computing for the Ascend AI chip.
<!-- Slide number: 45 -->
Overview of AI Chips
Hardware Architecture of Ascend Chips
Software Architecture of Ascend Chips
Logic Architecture of Ascend 310
Neural Network Software Flow of Ascend 310
Huawei Atlas AI Computing Platform
Industry Applications of Atlas
Notes:
<!-- Slide number: 46 -->
Neural Network Software Flow of Ascend AI Processors
The neural network software flow of Ascend AI processors is a bridge between the deep learning framework and Ascend AI chips. It realizes and executes a neural network application and integrates the following functional modules.
Process orchestrator: implements the neural network on Ascend AI chips, coordinates the whole process of effecting the neural network, and controls the loading and execution of offline models.
Digital vision pre-processing (DVPP) module: performs data processing and cleaning before input to meet format requirements for computing.
Tensor boosting engine (TBE): functions as a neural network operator factory that provides powerful computing operators for neural network models.
Publicité
Framework manager: builds an original neural network model into a form supported by Ascend AI chips, and integrates the new model into Ascend AI chips to ensure efficient running of the neural network.
Runtime manager: provides various resource management paths for task delivery and allocation of the neural network.
Notes:
<!-- Slide number: 47 -->
Neural Network Software Flow of Ascend AI Processors
Original Neural Network Model
Data
Framework
TBE
Generate
Offline Model
DVPP
Framework
Load and Execute
Runtime
Process Choreographer Coordination (Matrix)
TS
Task
Ascend AI Processor
Notes:
<!-- Slide number: 48 -->
Overview of AI Chips
Hardware Architecture of Ascend Chips
Software Architecture of Ascend Chips
Huawei Atlas AI Computing Platform
Industry Applications of Atlas
Notes:
<!-- Slide number: 49 -->
This section describes the main products of Huawei Atlas AI computing platform, including inference and training.
Notes:
<!-- Slide number: 50 -->
Atlas AI Computing Platform Portfolio
Internet, security, finance, transportation, power, etc.
Application
enabling
Cluster management/model management/data pre-processing
Industry SDK/container engine/basic service repository
Public components
Unified O&M and configuration management subsystem
Safety subsystem
AXE toolchain (log/profiling/Mind Studio)
AI framework
MindSpore
TensorFlow/PyTorch/Caffe/MxNet
Framework Adapter
Framework adapter
AscendCL
CANN
Graph optimization engine
Operator/Acceleration/Communication libraries (BLAS, FFT, DNN, Rand, Solver, Sparse, HCCL)
Runtime
Driver
Chips & Hardware
Atlas 800 AI inference server
512 TOPS INT8
Atlas 300 inference accelerator card
64 TOPS INT8
Atlas 200
16 TOPS INT8
Atlas 500
16 TOPS INT8
Atlas 900
256–1024 PFLOPS FP16
Atlas 800 AI training server
2 PFLOPS FP16
Atlas 300 training card
256 TFLOPS FP16
Atlas 200 developer kit
Da Vinci Architecture
Ascend 310
Ascend 910
Notes:
<!-- Slide number: 51 -->
Atlas Accelerates AI Inference
Ascend 310
AI processor
Edge intelligence and cloud-edge collaboration
Highest density in the industry (64-channel)
for video inference
Performance improved 7x for terminal devices
Powerful computing platform for AI inference
Atlas 800 AI server
Model: 3000/3010
Atlas 500 AI edge station
Model: 3000
Atlas 200 Developer Kit (DK) AI developer kit
Model: 3000
Atlas 300 AI accelerator card
Model: 3000
Atlas 200 AI accelerator module
Model: 3000
Notes:
<!-- Slide number: 52 -->
Atlas 200DK: Strong Computing Power and Ease-of-Use
16TOPS INT8 24W
1 USB type-C, 2 camera ports, 1 GE port, 1 SD card slot
8 GB memory
Operating temperature: 0°C to 45°C
Dimensions (H x W x D): 24 mm x 125 mm x 80 mm
Full-Stack AI development on and off the cloud
Researchers
Developers
Startups
Set up a dev environment with one laptop
Ultra low cost for local independent environment, with multiple functions and interfaces to meet basic requirements
Local dev + cloud training collaboration
Same protocol stack for Huawei Cloud and the developer kit; training on the cloud and deployment at local; no modification r...