Basic Information Page


My Credentials Page


Create Access Key

Configuring global settings for ModelArts: Go to the Settings page of ModelArts, and enter the AK and SK information recorded in the downloaded AK/SK file to authorize ModelArts modules to access OBS.


ModelArts Settings Page

Flower Recognition Application

The ExeML page consists of two parts. The upper part lists the supported ExeML project types. You can click Create Project to create an ExeML project. The created ExeML projects are listed in the lower part of the page. You can filter the projects by type or search for a project by entering its name in the search box and clicking.

The procedure for using ExeML is as follows:

Creating a project: To use ModelArts ExeML, create an ExeML project first.

Labeling data: Upload images and label them by class.

Training a model: After data labeling is completed, you can start model training.

Deploying a service and performing prediction: Deploy the trained model as a service and perform online prediction.

Creating a Project

Create a project.

On the ExeML page, click Create Project in Image Classification. The Create Image Classification Project page is displayed.


ModelArts ExeML


Create Image Classification Project

Parameters:

Billing Mode: Pay-per-use by default

Name: The value can be modified as required.

Description: The value can be modified as required.

Dataset Name: The value can be modified as required.

Input Dataset Path: Select an OBS path for storing the dataset to be trained. Create an empty folder on OBS first (Click the bucket name to enter the bucket. Then, click Create Folder, enter a folder name, and click OK). Select the newly created OBS folder as the training data path. Alternatively, you can import required data to OBS in advance. In this example, the data is uploaded to the /modelarts-demo/auto-learning/image-class folder.

For details about how to upload data, see: <https://support.huaweicloud.com/en-us/modelarts_faq/modelarts_05_0013.html>.

To obtain the source data, visit: <https://data-certification.obs.cn-east-2.myhuaweicloud.com/ENG/HCIA-AI/V3.0/modelarts-demo.rar>

Output Dataset Path: Select an OBS path for storing the dataset to be exported. Create an empty folder on OBS, the method of creation is the same as Input Dataset Path.

Confirm the project creation.

Click Create Project. The ExeML project is created.

Labeling Data

Upload images.

After an ExeML project is created, the Label Data page is automatically displayed.

Click Add Image to add images in batches. The dataset path is modelarts-datasets-and-source-code/ExeML/flower-recognition-application/training-dataset. If the images have been uploaded to OBS, click Synchronize Data Source to synchronize the images to ModelArts.


Upload Images


The images to be trained must be classified into at least two classes, and each class must contain at least five images. That is, at least two labels are available and the number of images for each label is not fewer than five.

You can add multiple labels to an image.

Label the images.

In area 1, click Unlabeled, and select one or more images to be labeled in sequence, or select Select Current Page in the upper right corner to select all images on the current page. In area 2, input a label or select an existing label and press Enter to add the label to the images. Then, click OK. The selected images are labeled.


Label the Images

Delete or modify a label in one image.

Click Labeled in area 1, and then click an image. To modify a label, click on the right of the label in area 2, enter a new label on the displayed dialog box, and click. To delete a label, click  on the right of the label in area 2. See Figure 1-10.


Delete or Modify Label in One Image

Delete or modify a label in multiple images.

In area 2, click the label to be modified or deleted, and click  on the right of the label to rename it, or click  to delete it from multiple images. In the dialog box that is displayed, select Delete label or Delete label and images that only contain this label.


Delete or Modify a Label in Multiple Images

Training a Model

After labeling the images, you can train an image classification model. Set the training parameters first and then start automatic training of the model. Images to be trained must be classified into at least two classes, and each class must contain at least five images. Therefore, before training, ensure that the labeled images meet the requirements. Otherwise, the Train button is unavailable.

Set related parameters.

You can retain the default values for the parameters, or modify Max Training Duration (h) and enable Advanced Settings to set the inference duration. Figure 1-12 shows the training settings.


Training Configuration

Parameters:

Max Training Duration (h): If the training process is not completed within the maximum training duration, it is forcibly stopped. You are advised to enter a larger value to prevent forcible stop during training.

Max Inference Duration (ms): The time required for inferring a single image is proportional to the complexity of the model. Generally, the shorter the inference time, the simpler the selected model and the faster the training speed. However, the precision may be affected.

Train a model.

After setting the parameters, click Train. After training is completed, you can view the training result on the Train Model tab page.

Deploying a Service and Performing Prediction

Deploy the model as a service.

After the model training is completed, you can deploy a version with the ideal precision and in the Successful status as a service. To do so, click Deploy in the Version Manager pane of the Train Model tab page. See Figure 1-13. After the deployment is successful, you can choose Service Deployment > Real-Time Services to view the deployed service.


Train Model

Test the service.

After the model is deployed as a service, you can upload an image to test the service. The path of the test data is modelarts-datasets-and-source-code/ExeML/flower-recognition-application/test-data/daisy.jpg.

On the Deploy Service tab page, click the Upload button to select the test image. After the image is uploaded successfully, click Predict. The prediction result is displayed in the right pane. See Figure 1-14. Five classes of labels are added during data labeling: tulip, daisy, sunflower, rose, and dandelion. The test image contains a daisy. In the prediction result, "daisy" gets the highest score, that is, the classification result is "daisy".


Deploy Service

Image Classification

Introduction

About This Experiment

This experiment is about a basic task in computer vision, that is image recognition. This experiment has been completed in chapter 4.Now, we will do this experiment on ModelArts with GPU.

Objectives

Upon completion of this task, you will be able to:

Know how to use ModelArts to develop your own model

Understand the advantages of deep neural network training with GPU acceleration

Procedure

Create a Notebook

Create a Notebook

Open the ModelArts Homepage and select DevEnviron to create Notebook.


ModelArts Homepage

Click the Create button to Create the Notebook environment.


Create Notebook

Configuration

Clicking the Create button, then you got the configuration page.Here are some parameters, explained as follows:

Name: user-defined.

Auto Stop: user-defined, it is recommended to estimate the usage time in advance.

Work Environment: Here, TF2.1 is fixed, as shown in the figure 2-3.

Instance Flavor: GPU: 1xV100 is recommended.


Notebook Configuration

Finish

Then, at the bottom of this page, click Next button to finish creating notebook.

Open Notebook and Coding

Once the Notebook is created, the Notebook environment that you created will be listed on this page. The Notebook environment can be managed with the Open, Stop, and Delete buttons. Click Open to continue.


Open Notebook

This interface is a Notebook page, click the New button, and then click TensorFlow-2.1.0 to create a New file, as shown on this figure.


Create a New File

This page is a Notebook's code editing interface and its base environment is TensorFlow2.1.0. You can start editing the code.


Notebook's Code Editing Interface

Experiment Code

Introducing Dependencies

Code:

import imageio

import numpy as np

import pickle

import os

import matplotlib.pyplot as plt

import tensorflow as tf

from tensorflow import keras

from tensorflow.keras import layers, optimizers, datasets, Sequential

from tensorflow.keras.layers import Conv2D,Activation,MaxPooling2D,Dropout,Flatten,Dense

Data Preprocessing

Code:

!wget <https://data-certification.obs.cn-east-2.myhuaweicloud.com/ENG/HCIA-AI/V3.0/cifar-10-python.tar.gz>

!tar -zxvf cifar-10-python.tar.gz

Code：

final\_tr\_data=[]

final\_tr\_label=[]

final\_te\_data=[]

final\_te\_label=[]

Decompress, return the decompressed dictionary

def unpickle(file):

fo = open(file, 'rb')

dict = pickle.load(fo, encoding='latin1')

fo.close()

return dict

Generate training set pictures. If you need png format, you only need to change the picture suffix name.

for j in range(1, 6):

dataName = "cifar-10-batches-py//data\_batch\_" + str(j) # Read the data\_batch12345 file in the current directory. The dataName is actually the path of the data\_batch file. The text and the script file are in the same directory.

Xtr = unpickle(dataName)

print(dataName + " is loading...")

for i in range(0, 10000):

img = np.reshape(Xtr['data'][i], (3,32, 32)) # Xtr['data'] is picture binary data

img = img.transpose(1, 2, 0)

picName = 'train//' + str(Xtr['labels'][i]) + '\_' + str(i + (j - 1)\*10000) + '.jpg' # Xtr['labels'] is the label of the picture, the value range is 0-9. In this article, the train folder needs to exist and be in the same directory as the script file.

if not os.path.exists('./train'):

os.mkdir('./train')

imageio.imwrite(picName, img)

final\_tr\_data.append(img)

final\_tr\_label.append(Xtr['labels'][i])

print(dataName + " loaded.")

print("test\_batch is loading...")

Generate test set images

testXtr = unpickle("cifar-10-batches-py//test\_batch")

for i in range(0, 10000):

img = np.reshape(testXtr['data'][i], (3,32, 32))

img = img.transpose(1, 2, 0)

picName = 'test//' + str(testXtr['labels'][i]) + '\_' + str(i) + '.jpg'

if not os.path.exists('./test'):

os.mkdir('./test')

imageio.imwrite(picName, img)

final\_te\_data.append(img)

final\_te\_label.append(testXtr['labels'][i])

print("test\_batch loaded.")

Output：


Code：

final\_tr\_data=np.array(final\_tr\_data)

print(np.shape(final\_tr\_data))

y\_train=np.array(final\_tr\_label).reshape(50000,1)

print(np.shape(final\_tr\_label))

final\_te\_data=np.array(final\_te\_data)

print(np.shape(final\_te\_data))

y\_test=np.array(final\_te\_label).reshape(10000,1)

print(np.shape(final\_te\_label))

Output：


Show the first 9 images:

Code:

category\_dict = {0:'airplane',1:'automobile',2:'bird',3:'cat',4:'deer',5:'dog',

6:'frog',7:'horse',8:'ship',9:'truck'}

#Show the first 9 images and their labels

plt.figure()

for i in range(9):

#create a figure with 9 subplots

plt.subplot(3,3,i+1)

#show an image

plt.imshow(final\_tr\_data[i])

#show the label

plt.ylabel(category\_dict[final\_tr\_label[i]])

plt.show()

#Pixel normalization

x\_train = final\_tr\_data.astype('float32')/255

x\_test = final\_te\_data.astype('float32')/255

num\_classes=10

Output:


First 9 Images with Tags

Model Creation

Code:

def CNN\_classification\_model(input\_size = x\_train.shape[1:]):

model = Sequential()

#the first block with 2 convolutional layers and 1 maxpooling layer

'''Conv1 with 32 3\*3 kernels

padding="same": it applies zero padding to the input image so that the input image gets fully covered by the filter and specified stride.

It is called SAME because, for stride 1 , the output will be the same as the input.

output: 32\32\32'''

model.add(Conv2D(32, (3, 3), padding='same',

input\_shape=input\_size))

#relu activation function

model.add(Activation('relu'))

#Conv2

model.add(Conv2D(32, (3, 3)))

model.add(Activation('relu'))

#maxpooling

model.add(MaxPooling2D(pool\_size=(2, 2),strides =1))

#the second block

model.add(Conv2D(64, (3, 3), padding='same'))

model.add(Activation('relu'))

model.add(Conv2D(64, (3, 3)))

model.add(Activation('relu'))

#maxpooling.the default strides =1

model.add(MaxPooling2D(pool\_size=(2, 2)))

#Before sending a feature map into a fully connected network, it should be flattened into a column vector.

model.add(Flatten())

#fully connected layer

model.add(Dense(128))

model.add(Activation('relu'))

#dropout layer.every neuronis set to 0 with a probability of 0.25

model.add(Dropout(0.25))

model.add(Dense(num\_classes))

#map the score of each class into probability

model.add(Activation('softmax'))

opt = keras.optimizers.Adam(lr=0.0001)

model.compile(loss='sparse\_categorical\_crossentropy', optimizer=opt, metrics=['accuracy'])

return model

model=CNN\_classification\_model()

model.summary()

Output:


Model Training

Code:

from tensorflow.keras.callbacks import ModelCheckpoint

model\_name = "final\_cifar10.h5"

model\_checkpoint = ModelCheckpoint(model\_name, monitor='loss',verbose=1, save\_best\_only=True)

#load pretrained models

trained\_weights\_path = 'cifar10\_weights.h5'

if os.path.exists(trained\_weights\_path):

model.load\_weights(trained\_weights\_path, by\_name =True)

#train

model.fit(x\_train,y\_train, batch\_size=32, epochs=10,callbacks = [model\_checkpoint],verbose=1)

Output:


This experiment is performed on a laptop. The network in this experiment is simple, consisting of four convolutional layers. To improve the performance of this model, you can increase the number of epochs and the complexity of the model.

Model Evaluation

Code:

new\_model = CNN\_classification\_model()

new\_model.load\_weights('final\_cifar10.h5')

model.evaluate(x\_test, y\_test, verbose=1)

Output:

10000/10000 [==============================] - 1s 66us/sample - loss: 0.8327 - accuracy: 0.7134

[0.8326702466964722, 0.7134]

Predict on a single image.

Code:

#output the possibility of each class

new\_model.predict(x\_test[0:1])

Output:

array([[2.3494475e-03, 6.9919275e-05, 8.1065837e-03, 7.8556609e-01,

2.3783690e-03, 1.8864134e-01, 6.8611270e-03, 1.2157968e-03,

4.3428279e-03, 4.6843957e-04]], dtype=float32)

Code:

#output the predicted label

new\_model.predict\_classes(x\_test[0:1])

Output:

array([3])

Plot the first 4 images in the test set and their corresponding predicted labels.

Code:

#label list

pred\_list = []

plt.figure()

for i in range(0,4):

plt.subplot(2,2,i+1)

#plot

plt.imshow(x\_test[i])

#predict

pred = new\_model.predict\_classes(x\_test[0:10])

pred\_list.append(pred)

#Display actual and predicted labels of images

plt.title("pred:"+category\_dict[pred[i]]+" actual:"+ category\_dict[y\_test[i][0]])

plt.axis('off')

plt.show()

Output:


The First 4 Images with Predicted Labels

Summary

This chapter describes how to build an image classification model based on ModelArts.