Graph neural networks:

Combine NN and graph structure data.

AI solution

Design new solutions with the help of neural networks.

Framework resolution

Accelerate equation solving with the help of new frameworks.

AI modeling

Build AI-based computable models.

Challenges:

Integrate NN models and probability models for modeling, reducing the learning difficulty.

Store, partition, and sample trillions of distributed graph data.

Support dynamic network structure and elastically distributed training.

Challenges:

Equations as code. Users can quickly construct expressions, and the serial coding and parallel coding are consistent.

Support large-scale heterogeneous parallelism and mixed precision computing.

Support high-performance higher-order differentiation (the volume of computing higher-order differentiation increases exponentially with the order).

Computing graphs (Taichi)

Differentiable physical engines

Challenges:

Sparse expression

Separation of data and computing

Differentiable programming

Notes:

Why did we develop MindSpore? There are two reasons:

1. In the context of the technical and trade wars, Huawei must have an independent and controllable computing framework for all scenarios.

2. The current computing framework still has a large number of problems to be urgently resolved, such as high entry barriers, high execution costs, and great deployment difficulty.

MindSpore engineers hope to boost inclusive AI. In this regard, we propose three points for technical innovations: new programming paradigm, execution mode, and collaboration method.

<!-- Slide number: 11 -->

Challenge 3: Continuously Increasing Computing Power and Complexity

Key challenges to AI framework software during AI chip development:

Improve the coupling of optimization and hardware, and integrate graph build and operator build.

Fusion optimization at graph layer: Converge hardware-independent optimization, fully utilize the hardware computing power, and break the boundary between the subgraph level and operator level for overall optimization.

Optimization at operator layer: Consider hardware capabilities when using operators to implement algorithms.

Apply model execution modes to scenarios and hardware.

Mix the graph sink mode and single-operator execution. Use different optimal mode according to the hardware.

Use the data flow execution mode to better exert the computing power.

Use the SoC-level distributed parallel strategy for packaging more silicon chips.

Use virtualization execution mode in SoC.

Huge programmability challenges.

The effective computing power is close to the theoretical computing power and has high requirements on the compiler;

Sparse acceleration, image preprocessing acceleration module, and complex SIMD acceleration instructions;

SoC-level heterogeneous programming: CUBE core, Vector core, and ARM.

Multi-chip, single-chip cross-generation, and cross-model compatibility requirements.

Mainstream hardware

GPU: NVIDIA commercial product

NPU: Huawei commercial product

TPU: Google commercial product

Hardware development trends

Increase the computing density of a single core, improve the bandwidth and the process, increase the number of cores, and package more silicon chips.

Widely use SIMD and increase the tensor core processing scale (4 x 4 → 16 x 16).

New data types (such as TF32 and BF16), high-speed interconnection between chips, and support for virtualization.

Notes:

Why did we develop MindSpore? There are two reasons:

1. In the context of the technical and trade wars, Huawei must have an independent and controllable computing framework for all scenarios.

2. The current computing framework still has a large number of problems to be urgently resolved, such as high entry barriers, high execution costs, and great deployment difficulty.

MindSpore engineers hope to boost inclusive AI. In this regard, we propose three points for technical innovations: new programming paradigm, execution mode, and collaboration method.

<!-- Slide number: 12 -->

Challenge 4: New Programming Languages Making Breakthroughs in Python

Func f(_ x:Float) ->Float{ return tanh(log(x) }

#gradient(of: f)

FLUX

Julia Src

Julia AST

Julia IR & optimization

Zygote

Auto differentiation

CUBlas

XLA

LLVM IR & optimization

Executable program

Lowering

SIL Mandatory Passes

Reverse-mode AD

Primal Generation

Adjoint Generation

Sil @f : $(Float) -> Float

Sil @f__primal : $(Float) -> f_Checkpoints

Sil @f__adjoint : $(Float, f__Checkpoint, Float) ->Float

Sil @f__gradient : $(Float) -> Float

Julia enters the AI field based on the tensor native expression, IR openness, and high performance as well as the accumulation in the scientific computing and HPC fields.

Swift for TensorFlow tries to find differentiated competitiveness based on enterprise-class features such as static type, easy deployment, and high performance.

Notes:

Why did we develop MindSpore? There are two reasons:

1. In the context of the technical and trade wars, Huawei must have an independent and controllable computing framework for all scenarios.

2. The current computing framework still has a large number of problems to be urgently resolved, such as high entry barriers, high execution costs, and great deployment difficulty.

MindSpore engineers hope to boost inclusive AI. In this regard, we propose three points for technical innovations: new programming paradigm, execution mode, and collaboration method.

<!-- Slide number: 13 -->

Challenge 5: Unification of Dynamic and Static Graphs


Research phase: dynamic graph; Python affinity, flexibility, and usability.

Production phase: static graph; performance, and deployment capability.

Pain point: The representation of the dynamic graph is not completely the same as that of the static graph.

Trend: Optimize JIT to achieve the consistency of the two representations.

Challenge: It is difficult to fully use JIT to support Python flexibility and dynamics.

Industry frameworks use compilers such as accelerated linear algebra (XLA) to work with chips for in-depth optimization.

Gradually improve the IR from the perspective of optimization to form open AI infrastructure, such as Relay/TVM and MLIR.

Notes:

Why did we develop MindSpore? There are two reasons:

1. In the context of the technical and trade wars, Huawei must have an independent and controllable computing framework for all scenarios.

2. The current computing framework still has a large number of problems to be urgently resolved, such as high entry barriers, high execution costs, and great deployment difficulty.

MindSpore engineers hope to boost inclusive AI. In this regard, we propose three points for technical innovations: new programming paradigm, execution mode, and collaboration method.

<!-- Slide number: 14 -->

Challenge 6: AI Deployment in All Scenarios

According to the 2019 CIO Agenda survey conducted by Gartner, the proportion of enterprises that have deployed AI increased from 4% to 14% from 2018 to 2019. The data is in sharp contrast to the industry's increasing awareness of the value of AI.

Trend 1: To reduce latency and improve user experience, on-device language model deployment becomes a trend. The challenge is how to reduce the model size and minimize the precision loss.

Cloud AI


Trend 2: device-cloud synergy

Mobile AI = On-Device AI + Smart services, better considering personalization, security, and privacy.

Single agent → multiple agent collaboration, implementing real-time perception and decision-making.

Privacy and security

Low latency

High reliability

Low bandwidth required

High computing power

Large model

Big data

High network bandwidth

Trend 3: Ubiquitous AI is deployed in scenarios where IoT and smart devices have extremely limited resources.



Tapping mode identification:

Deep learning (90%) vs. Traditional method (60%)

Notes:

Why did we develop MindSpore? There are two reasons:

1. In the context of the technical and trade wars, Huawei must have an independent and controllable computing framework for all scenarios.

2. The current computing framework still has a large number of problems to be urgently resolved, such as high entry barriers, high execution costs, and great deployment difficulty.

MindSpore engineers hope to boost inclusive AI. In this regard, we propose three points for technical innovations: new programming paradigm, execution mode, and collaboration method.

<!-- Slide number: 15 -->

Challenge 7: Security, Privacy, and Protection

Security

Transparency

Privacy

Explainability

Fairness

Responsible AI

Adversarial examples

Model theft

Model backdoor

Model inversion

Encrypted AI

Federated learning

Comprehensibility

Accountability

Individual fairness

Group fairness

Key challenges:

There is no general analysis method and measurement system for all aspects of responsible AI, and there is no automatic measurement method for scenario awareness.

AI model robustness, privacy protection technologies, and encrypted AI have great impact on model performance in actual scenarios.

Responsible AI is deeply combined with AI explainability and verifiability.

Trend insights:

In the future, in addition to accuracy and performance, meeting responsible AI will be a key requirement for AI service success.

The AI framework bears AI services and must have the capability of enabling responsible AI.

Notes:

Why did we develop MindSpore? There are two reasons:

1. In the context of the technical and trade wars, Huawei must have an independent and controllable computing framework for all scenarios.

2. The current computing framework still has a large number of problems to be urgently resolved, such as high entry barriers, high execution costs, and great deployment difficulty.

MindSpore engineers hope to boost inclusive AI. In this regard, we propose three points for technical innovations: new programming paradigm, execution mode, and collaboration method.

The EU establishes responsible AI through technical specifications.

The US is actively researching and regulating investment in the responsible AI field.

Google+DeepMind advocate responsible AI and provides comprehensive technical support.

Microsoft plans six attributes of responsible AI and continues research.

2019: Digital ethics and privacy -> 2020: AI security, transparency, and traceability. Gartner predicts that by 2022, 30% cyber attacks will use training data to poison, AI models to steal, or adversarial example technologies.

<!-- Slide number: 16 -->

AI Framework Development Trends and Challenges

MindSpore Development Framework

MindSpore Architecture

MindSpore Key Features

MindSpore Development and Application

Notes:

<!-- Slide number: 17 -->

MindSpore Open-source Deep Learning Framework

MindSpore

MindSpore Lite

MindExpress

Lightweight inference tool

Ultra-lightweight inference

On-device learning

Cangjie/Julia language

ModelZoo

ModelZoo

Incremental backup/transfer learning

NN

Operator

Data

Inference

Training

Model format conversion

Federated learning

Quantization (int8/int4/mixed precision)

Heterogeneous scheduling

Memory overcommitment

Data processing

Type derivation

Automatic differentiation

Auto parallelism

Second-order optimization

Graph kernel fusion

Memory optimization

MindArmour

MindArmour

Generating graph operator code

Custom fine-tune layer

Knowledge distillation/pruning

Online subgraph partition

Kernel Select

Integrated MindIR

MindData

MindData

GE (accelerated by Ascend)

MindAKG

MindRT Lite

Multiple batches/sessions

Heterogeneous execution

Operator synchronization/asynchronization

High-performance memory allocation

Large and small core scheduling

MindRT

Distributed DAG parallel execution

High-performance CPU/GPU operator library and Kirin/MTK AI acceleration library

Superior performance

Lightweight

All-scenario support

Efficient deployment

Notes:

<!-- Slide number: 18 -->

MindSpore Vision and Value

Lower the barrier for AI development, maximize Ascend computing power, and empower inclusive AI.

Efficient development

Profound expertise required

Algorithms

Programming


High performance

Flexible deployment

High deployment time and cost

Develop once

Deploy everywhere

Diverse computing power and models

CPU+NPU

Graph+Matrix

Notes:

<!-- Slide number: 19 -->

MindSpore Logical Architecture

Design Objectives

Beyond AI: NN applications  general AI + numerical computation

Key feature: general-purpose tensor derivable computing

Distributed parallel native: supporting AI models to go beyond trillions of parameters

Key features: automatic parallelism, memory-constrained programming, and second-order optimization

In-depth graph kernel fusion: capitalizing on the computing power of AI chips

Key features: joint graph and kernel optimization as well as automatic optimization based on Poly

Enterprise-level capabilities in all scenarios: flexible deployment and collaboration, secure, reliable, and explainable

Key features: ultra-lightweight runtime, private training, adaptive model generation, quantitative training, and explainable AI

Design philosophy: AI "JDK"

Representation/optimization/operation decoupling: multi-frontend, cross-chip, and cross-platform

Openness: opening the general graph compilation and running capabilities to third-party frameworks

Centralized architecture for all scenarios: integrated APIs and IRs, enabling smooth AI applications

MindSpore Extend

GNN, deep probabilistic programming, reinforcement learning, differential equation, and more

MindExpress

Model Zoo

Mind

Armour

Encrypted AI, model obfuscation, and device-cloud synergy for privacy protection

Cangjie, Julia, 3rd frontend

NN

Data

Operator

Training

Inference

Mind

Data

MindCompiler

Quantization, pruning, and more

MindIR

MindAKG (poly automatic optimization)

Data processing

Type derivation

Automatic differentiation

Auto parallelism

Second-order optimization

Mind

Insight

Memory optimization

Graph kernel fusion

Pipeline execution

Debug and tuning

MindRT

MindRT

MindRT Lite/Macro

Distributed heterogeneous parallel

Operator

Heterogeneous parallel

Operator

CUDA

Eigen...

CANN (Ascend)

Android, Harmony, IoT

Notes:

<!-- Slide number: 20 -->

Subsystem: MindExpress

Design objectives:

Design both high-level and low-level APIs for users, supporting network building, entire graph execution, subgraph execution, and single-operator execution.

Provide integrated APIs for model training, inference, and export, suitable for various scenarios, such as the device, edge, and cloud.

Provide unified encoding for dynamic and static graphs.

Provide unified encoding for standalone and distributed training.

High-Level Python API

Callback

Quant

Model

Amp

Parallel

Other

Cell

Tensor

Low-Level Python API

Layers

Metric

Optimizer

Loss

Functional modules:

High-level APIs provide management, callback, quantization, mixed precision, and parallel control APIs for training and inference, facilitating process control on the entire network.

Low-level APIs provide basic tensors, cells, NN-layers, optimizers, and initialization, helping users flexibly build networks and control execution processes.

The executor controls computing execution and interacts with the MindSpore backend.

Utils

Initializer

Ops

Other

Executor

Notes:

<!-- Slide number: 21 -->

Subsystem: MindCompiler

MindCompiler provides the just-in-time compilation capability for MindIR.

Graph high level optimization (GHLO) is application-oriented and provides frontend optimization and functions, such as Type derivation, automatic differentiation, second-order optimization, and automatic parallelism.

Graph low level optimization (GHLO) is hardware-oriented and performs bottom-layer optimization, such as operator fusion, layout optimization, redundancy elimination, and memory optimization.

Mixed precision

Type derivation

Automatic differentiation

Auto parallelism

Second-order optimization

Graph kernel fusion

MindCompiler

C ++ API

Backends: Ascend, GPU, and more

GHLO

Operator selection

Layout optimization

Memory optimization

GLLO

MindIR

Notes:

<!-- Slide number: 22 -->

Subsystem: MindRT

The centralized runtime system supports:

Multiple device types on the device and cloud

Scheduling management of multiple hardware platforms, such as Ascend, GPU, and CPU

Memory pooling management and efficient memory overcommitment

Asynchronous operators, heterogeneous execution, and multi-flow concurrency

MindRT

MindRT

MindRT lite

ops

ops

runtime

runtime

host

Ascend

Entire graph execution

Technology features

Input and output data is transferred through cache queues, and the zero copy mechanism ensures that data copies are fully hidden.

The entire graph is offloaded to avoid extra host-device interaction overheads.

Ascend

Notes:

<!-- Slide number: 23 -->

#

Subsystem: MindData

MindData is responsible for efficiently executing the training data processing pipeline, forming a pipeline with computing, and promptly importing data for training.

load

shuffle

map

batch

repeat

Typical training data processing pipeline

Key functions:

Pipeline + parallel execution, improving data processing throughput

Various data operators

User-defined Python operators and pipelines (data loading, sampling, and argumentation)

Heterogeneous hardware acceleration (Ascend/GPU/CPU)

MindRecord: built-in metadata and aggregated storage

MindData

API(Python/C++)

C++ core

Data graph generation

Data graph execution

Data operators (loading/argumentation/sending)

MindRecord

/TFRecord/Other

Ascend/GPU/CPU

Running process:

Data graph generation: Data graphs are generated based on Python APIs called by users.

Data graph execution: The pipeline executes data operators in a data graph; this happens in parallel to complete dataset loading, shuffle, data argumentation, and batch processing.

Importing data to device: The processed data is imported to the device for training.

Notes:

<!-- Slide number: 24 -->

Subsystem: MindInsight

MindInsight is the debugging and optimization subsystem of MindSpore. It provides the training process visualization, model lineage, debugger, and performance profiling functions.

MindInsight UI

MindInsight backend

Key functions:

APIs are easy to use, enabling users to easily collect training process metrics, including computational graphs, scalar data (such as loss and accuracy), histogram data (such as gradient and weight), and performance data, and display them on the web UI.

Collect training hyperparameters, datasets, and data augmentation information to implement model lineage and compare training results.

RESTful API

Training log parsing

RPC communication

Visualization

Hyperparameter lineage

Performance profiling

Debugger

Running process:

Collecting training information: Users collect common training indicators using the callback API, and can decide which information to collect based on their requirements. For example, use the summary operator to collect information about the computational graph and the Python API for information about the Python layer.

Generating training logs: Training logs are generated based on the process information collected during training.

Displaying training information: MindInsight opens and parses training logs to display the training process information in a graph.

MindSpore

Training information collection APIs

Training information exchange

Summary operator

Python API

Summary file

RPC

Information collection callback

Writer pool

FP/BP loop

Notes:

<!-- Slide number: 25 -->

Subsystem: MindArmour

MindArmour provides comprehensive, effective, and easy-to-use evaluation tools and enhancement methods for AI trustworthiness in each domain.

Trustworthy AI

Security

Privacy

Explainability

Fairness

Transparency

Start

We are HERE

Key functions:

Test data generation methods in all scenarios, such as black-and-white box adversarial attacks, member and attribute inference attacks, and data drifts.

Coverage-based fuzzing test process as well as flexible and customizable test strategies and indicators.

Common methods of adversarial example detection and model robustness enhancement, including adversarial training and input rebuilding.

Efficient, adaptive, and differential privacy training and budget statistics algorithms, as well as mathematically proven model for privacy leakage constraints.

Goal

AI Model Trustworthiness Test

Model Evaluation

Fuzzer

Data/Test Generator

Trustworthiness Evaluation

Adversarial Attack

Benign Transform

Coverage Evaluation

Running process:

Configuring strategies: Define test strategies based on threat vectors and trustworthiness requirements, and select the appropriate test data generation methods.

Executing fuzzing test: Generate trustworthiness test data heuristically based on the model coverage rate and configured strategies.

Generating evaluation reports: Generate said reports based on built-in or user-defined trustworthiness indicators.

Enhancing trustworthiness: Use preset methods to enhance the trustworthiness of AI models.

Attribute

Inference

Membership Inference

AI Trustworthiness Enhancement

Privacy Tools

Robustness Tools

Adversarial Training

Adversarial Training

Adversarial Detection

Adaptive Differential Privacy Training

Notes:

<!-- Slide number: 26 -->

AI Framework Development Trends and Challenges

MindSpore Development Framework

MindSpore Architecture

MindSpore Key Features

MindSpore Development and Application

Notes:

<!-- Slide number: 27 -->

MindSpore Feature: Automatic Parallelism

MindSpore Key Features

Challenges

Challenges to efficient distributed training of ultra-large models:

NLP models become larger and larger. The memory overhead for training ultra-large models such as BERT (340 million)/GPT-2 (1542 million) exceeds the capacity of a single device. Therefore, the model needs to be partitioned into multiple devices for execution.

Currently, manual model parallelism requires model partitioning design and cluster topology awareness, which is difficult to develop, and it is hard to ensure high performance and perform tuning.

Automatically partition an entire graph based on the input and output data of the operator, and integrate data parallelism and model parallelism. Cluster topology aware scheduling: The cluster topology is aware, and subgraphs are automatically scheduled to minimize communication costs.

NN Graph

Dense

MatMul

Subgraph 1

Dense

MatMul

Subgraph 2

MatMul

Dense

Network

CPU

Ascend

Ascend

CPU

Ascend

Ascend

Effect: The standalone code logic is kept to implement model parallelism, improving development efficiency by 10 times compared with manual parallelism!

Notes:

Efficient hybrid parallel of ultra-large models, and smooth expansion of computing power

<!-- Slide number: 28 -->

MindSpore Feature: Second-order Optimization

MindSpore Key Features

Challenges

The second-order matrix is approximated to reduce the computational complexity, and then the frequency and dimension of the matrix are reduced to accelerate the computation.


Parameter

First-order

gradient

Learning rate

Second-order

information matrix

| Optimizer | Epoch | Convergence Time | Test Scale |

| --- | --- | --- | --- |

| SGD+MOMENTUM | About 60 | 90 minutes | 8-device Ascend 910 |

| MindSpore second-order optimization | 42 | 71.5 minutes | 8-device Ascend 910 |

Core problem: The second-order optimizer needs to compute the inverse matrix of the second-order information matrix. The computation workload is heavy, and it can take hours to solve the second-order matrix directly, creating a technical difficulty.

Notes:

CNN training acceleration is supported.

The training convergence speed is accelerated by 20.6% based on ResNet-1.5@ImageNet2012.

<!-- Slide number: 29 -->

MindSpore Feature: On-Device Execution

MindSpore Key Features

Challenges

Challenges to model execution with powerful chip computing power:

Memory wall problems, high interaction overhead, and difficult data supply. Some operations are performed on the host, while others are performed on the device. The interaction overhead is much greater than the execution overhead. As a result, the accelerator usage is low.

The chip-oriented deep graph optimization is used to reduce synchronization waiting time and maximize the parallelism degree of "data-computing-communication". Data + Entire computational graph to the Ascend chips.



CPU

conv

conv

bn

relu6

add

conv

bn

relu6

dwconv

bn

relu6

GPU

Data copy

Conditional redirection task

Dependency Notification Task

kernel1

kernel2

...

Effect: Compared with the host-side graph scheduling mode, the training performance is improved by 10 times!

High data interaction overhead and difficult data supply.

Notes:

The entire graph is offloaded to devices, maximizing the computing power of Ascend.

In addition to programming, we have done a lot in execution. As our computing power becomes stronger, the control plane and data augmentation will become huge bottlenecks. For example, currently, data augmentation operators, hardwork operator libraries, and convolution and operator execution are all implemented based on the libraries. In addition, the software provides many operator fusion functions to further improve the performance. The overall performance is improved by 10 times compared with the original mode, not to mention the comparison with other vendors the industry.

<!-- Slide number: 30 -->

MindSpore Feature: Deployment and Collaboration in All Scenarios

MindSpore Key Features

Challenges

Unified model IR brings consistent deployment experience.

The graph optimization technology based on software and hardware collaboration shields scenario differences.

Federal meta learning based on device-cloud synergy breaks the boundaries of devices and the cloud. The multi-device collaboration model is updated in real time.

The diversity of hardware architectures leads to deployment differences and performance uncertainties in all scenarios, and the separation of training and inference results in model isolation.

Effect: In the unified architecture, the deployment performance of models in all scenarios is consistent, and the accuracy of personalized models is improved!

On-demand collaboration in all scenarios and consistent development experience

Edge

Device

Cloud




Notes:

Key technology: The IR of the unified model copes with the upper-layer differences in different language scenarios. The data structure is customized to ensure compatibility. Hardware-software synergy can also be achieved. Finally, the whole architecture is integrated, which is the most popular direction in the industry.

MindIR is an intermediate representation provided by MindSpore. It can help you implement multiple deployments in one training session and implement device-cloud interconnection. Simply put, you can train and generate MindIR models on the Ascend, GPU, and CPU hardware platforms, and then quickly deploy the models on mobile phones and inference servers to experience the full-scenario capabilities of MindSpore.

MindIR is a functional IR based on graph representation. It defines the extensible graph structure and the operator IRs, and stores the basic data structures of MindSpore, including computational graphs and parameter weights.

MindIR eliminates model differences between different backends through unified operator IR definitions. You can perform collaborative tasks on different platforms (device, edge, and cloud) based on the same model file.

<!-- Slide number: 31 -->

MindSporeIR

MindSporeIR...