#The most commonly used loss function of linear regression model is the loss function of mean variance difference

def loss\_function(a, b, x, y):

num = len(x)

prediction=model(a,b,x)

return (0.5/num) \* (np.square(prediction-y)).sum()

#The optimization function mainly USES partial derivatives to update two parameters a and b

def optimize(a,b,x,y):

num = len(x)

prediction = model(a,b,x)

#Update the values of A and B by finding the partial derivatives of the loss function on a and b

da = (1.0/num) \ ((prediction -y)\x).sum()

db = (1.0/num) \* ((prediction -y).sum())

a = a - Lr\*da

b = b - Lr\*db

return a, b

#iterated function, return a and b

def iterate(a,b,x,y,times):

for i in range(times):

a,b = optimize(a,b,x,y)

return a,b

Start the iteration

Initialization and Iterative optimization model

Code:

#Initialize parameters and display

a = np.random.rand(1)

print(a)

b = np.random.rand(1)

print(b)

Lr = 1e-4

#For the first iteration, the parameter values, losses, and visualization after the iteration are displayed

a,b = iterate(a,b,x,y,1)

prediction=model(a,b,x)

loss = loss\_function(a, b, x, y)

print(a,b,loss)

plt.scatter(x,y)

plt.plot(x,prediction)

Output:


Iterate 1 time

In the second iteration, the parameter values, loss values and visualization effects after the iteration are displayed

Code:

a,b = iterate(a,b,x,y,2)

prediction=model(a,b,x)

loss = loss\_function(a, b, x, y)

print(a,b,loss)

plt.scatter(x,y)

plt.plot(x,prediction)

Output:


Iterate 2 times

The third iteration shows the parameter values, loss values and visualization after iteration

Code:

a,b = iterate(a,b,x,y,3)

prediction=model(a,b,x)

loss = loss\_function(a, b, x, y)

print(a,b,loss)

plt.scatter(x,y)

plt.plot(x,prediction)

Output:


Iterate 3 times

In the fourth iteration, parameter values, loss values and visualization effects are displayed

Code:

a,b = iterate(a,b,x,y,4)

prediction=model(a,b,x)

loss = loss\_function(a, b, x, y)

print(a,b,loss)

plt.scatter(x,y)

plt.plot(x,prediction)

Output:


Iterate 4 times

The fifth iteration shows the parameter value, loss value and visualization effect after iteration

Code:

a,b = iterate(a,b,x,y,5)

prediction=model(a,b,x)

loss = loss\_function(a, b, x, y)

print(a,b,loss)

plt.scatter(x,y)

plt.plot(x,prediction)

Output:


Iterate 5 times

The 10000th iteration, showing the parameter values, losses and visualization after iteration

Code:

a,b = iterate(a,b,x,y,10000)

prediction=model(a,b,x)

loss = loss\_function(a, b, x, y)

print(a,b,loss)

plt.scatter(x,y)

plt.plot(x,prediction)

Output:


Iterate 10000 times

Thinking and practice

Question 1

Try to modify the original data yourself, Think about it: Does the loss value have to go to zero？

Question 2

Modify the values of Lr, Think: What is the role of the Lr parameter？

Decision tree details

Introduction

About This Experiment

This experiment focuses on the decision tree algorithm through the basic Python code.

It mainly uses Numpy module, Pandas module and Math module. We will implement the CART tree（Classification and Regressiontree models) in this experiment.

You have to download the dataset before this experiment through this link:

https://data-certification.obs.cn-east-2.myhuaweicloud.com/ENG/HCIA-AI/V3.0/ML-Dataset.rar

Objectives

The purpose of this experiment is as follows:

• Familiar with basic Python syntax
• Master the principle of Classification tree and implement with Python code
• Master the principle of Regression tree and implement with Python code

Experiment Code

Import the modules you need

Pandas is a tabular data processing module.

Math is mainly used for mathematical calculations.

Numpy is the basic computing module.

Code:

import pandas as pd

import math

import numpy as np

Superparameter definition section

Here you can choose to use Classification tree or Regression tree. Specifies the address of the dataset. Get feature name. Determine whether the algorithm matches the data set

Code:

algorithm = "Regression" # Algorithm: Classification, Regression

algorithm = "Classification" # Algorithm: Classification, Regression

Dataset1: Text features and text labels

#df = pd.read\_csv("D:/Code/Decision Treee/candidate/decision-trees-for-ml-master/decision-trees-for-ml-master/dataset/golf.txt")

Dataset2: Mix features and Numeric labels, here you have to change the path to yours.

df = pd.read\_csv("ML-Dataset/golf4.txt")

This dictionary is used to store feature types of continuous numeric features and discrete literal features for subsequent judgment

dataset\_features = dict()

num\_of\_columns = df.shape[1]-1

#The data type of each column of the data is saved for displaying the data name

for i in range(0, num\_of\_columns):

#Gets the column name and holds the characteristics of a column of data by column

column\_name = df.columns[i]

#Save the type of the data

dataset\_features[column\_name] = df[column\_name].dtypes

The size of the indent when display

root = 1

If the algorithm selects a regression tree but the label is not a continuous value, an error is reported

if algorithm == 'Regression':

if df['Decision'].dtypes == 'object':

raise ValueError('dataset wrong')

If the tag value is continuous, the regression tree must be used

if df['Decision'].dtypes != 'object':

algorithm = 'Regression'

global\_stdev = df['Decision'].std(ddof=0)

Define the functions required to complete the algorithm

ProcessContinuousFeatures: Used to convert a continuous digital feature into a category feature.

Code:

This function is used to handle numeric characteristics

def processContinuousFeatures(cdf, column\_name, entropy):

Numerical features are arranged in order

unique\_values = sorted(cdf[column\_name].unique())

subset\_ginis = [];

subset\_red\_stdevs = []

for i in range(0, len(unique\_values) - 1):

threshold = unique\_values[i]

Find the segmentation result if the first number is used as the threshold

subset1 = cdf[cdf[column\_name] <= threshold]

subset2 = cdf[cdf[column\_name] > threshold]

Calculate the proportion occupied by dividing the two parts

subset1\_rows = subset1.shape[0];

subset2\_rows = subset2.shape[0]

total\_instances = cdf.shape[0]

In the text feature part, entropy is calculated by using the cycle,

and in the numeric part, entropy is calculated by using the two groups after segmentation,

and the degree of entropy reduction is obtained

if algorithm == 'Classification':

decision\_for\_subset1 = subset1['Decision'].value\_counts().tolist()

decision\_for\_subset2 = subset2['Decision'].value\_counts().tolist()

gini\_subset1 = 1;

gini\_subset2 = 1

for j in range(0, len(decision\_for\_subset1)):

gini\_subset1 = gini\_subset1 - math.pow((decision\_for\_subset1[j] / subset1\_rows), 2)

for j in range(0, len(decision\_for\_subset2)):

gini\_subset2 = gini\_subset2 - math.pow((decision\_for\_subset2[j] / subset2\_rows), 2)

gini = (subset1\_rows / total\_instances) \ gini\_subset1 + (subset2\_rows / total\_instances) \ gini\_subset2

subset\_ginis.append(gini)

Take standard deviation as the judgment basis, calculate the decrease value of standard deviation at this time

elif algorithm == 'Regression':

superset\_stdev = cdf['Decision'].std(ddof=0)

subset1\_stdev = subset1['Decision'].std(ddof=0)

subset2\_stdev = subset2['Decision'].std(ddof=0)

threshold\_weighted\_stdev = (subset1\_rows / total\_instances) \* subset1\_stdev + (

subset2\_rows / total\_instances) \* subset2\_stdev

threshold\_reducted\_stdev = superset\_stdev - threshold\_weighted\_stdev

subset\_red\_stdevs.append(threshold\_reducted\_stdev)

#Find the index of the split value

if algorithm == "Classification":

winner\_one = subset\_ginis.index(min(subset\_ginis))

elif algorithm == "Regression":

winner\_one = subset\_red\_stdevs.index(max(subset\_red\_stdevs))

Find the corresponding value according to the index

winner\_threshold = unique\_values[winner\_one]

Converts the original data column to an edited string column.

Characters smaller than the threshold are modified with the <= threshold value

cdf[column\_name] = np.where(cdf[column\_name] <= winner\_threshold, "<=" + str(winner\_threshold),">" + str(winner\_threshold))

return cdf

CalculateEntropy: Used to calculate Gini or variances, they are the criteria for classifying.

Code:

This function calculates the entropy of the column, and the input data must contain the Decision column

def calculateEntropy(df):

The regression tree entropy is 0

if algorithm == 'Regression':

return 0

rows = df.shape[0]

Use Value\_counts to get all values stored as dictionaries, keys: finds keys, and Tolist: change to lists.

This line of code finds the tag value.

decisions = df['Decision'].value\_counts().keys().tolist()

entropy = 0

Here the loop traverses all the tags

for i in range(0, len(decisions)):

Record the number of times the tag value appears

num\_of\_decisions = df['Decision'].value\_counts().tolist()[i]

probability of occurrence

class\_probability = num\_of\_decisions / rows

Calculate the entropy and sum it up

entropy = entropy - class\_probability \* math.log(class\_probability, 2)

return entropy

FindDecision: Find which feature in the current data to classify.

Code:

The main purpose of this function is to traverse the entire column of the table,

find which column is the best split column, and return the name of the column

def findDecision(ddf):

If it's a regression tree, then you take the standard deviation of the true value

if algorithm == 'Regression':

stdev = ddf['Decision'].std(ddof=0)

Get the entropy of the decision column

entropy = calculateEntropy(ddf)

columns = ddf.shape[1];

rows = ddf.shape[0]

Used to store Gini and standard deviation values

ginis = [];

reducted\_stdevs = []

Traverse all columns and calculate the relevant indexes of all columns according to algorithm selection

for i in range(0, columns - 1):

column\_name = ddf.columns[i]

column\_type = ddf[column\_name].dtypes

Determine if the column feature is a number, and if so, process the data using the following function,

which modifies the data to a string type category on return.

The idea is to directly use character characteristics, continuous digital characteristics into discrete character characteristics

if column\_type != 'object':

ddf = processContinuousFeatures(ddf, column\_name, entropy)

The statistical data in this column can be obtained, and the continuous data can be directly classified after processing,

and the categories are less than the threshold and greater than the threshold

classes = ddf[column\_name].value\_counts()

gini = 0;

weighted\_stdev = 0

Start the loop with the type of data in the column

for j in range(0, len(classes)):

current\_class = classes.keys().tolist()[j]

The final classification result corresponding to the data is selected

by deleting the value of the df column equal to the current data

subdataset = ddf[ddf[column\_name] == current\_class]

subset\_instances = subdataset.shape[0]

The entropy of information is calculated here

if algorithm == 'Classification': # GINI index

decision\_list = subdataset['Decision'].value\_counts().tolist()

subgini = 1

for k in range(0, len(decision\_list)):

subgini = subgini - math.pow((decision\_list[k] / subset\_instances), 2)

gini = gini + (subset\_instances / rows) \* subgini

The regression tree is judged by the standard deviation,

and the standard deviation of the subclasses in this column is calculated here

elif algorithm == 'Regression':

subset\_stdev = subdataset['Decision'].std(ddof=0)

weighted\_stdev = weighted\_stdev + (subset\_instances / rows) \* subset\_stdev

Used to store the final value of this column

if algorithm == "Classification":

ginis.append(gini)

Store the decrease in standard deviation for all columns

elif algorithm == 'Regression':

reducted\_stdev = stdev - weighted\_stdev

reducted\_stdevs.append(reducted\_stdev)

Determine which column is the first branch

by selecting the index of the largest value from the list of evaluation indicators

if algorithm == "Classification":

winner\_index = ginis.index(min(ginis))

elif algorithm == "Regression":

winner\_index = reducted\_stdevs.index(max(reducted\_stdevs))

winner\_name = ddf.columns[winner\_index]

return winner\_name

FormatRule: Standardize the final output format.

Code:

ROOT is a number used to generate ' 'to adjust the display format of the decision making process

def formatRule(root):

resp = ''

for i in range(0, root):

resp = resp + ' '

return resp

BuildDecisionTree: Main function.

Code:

With this function, you build the decision tree model,

entering data in dataframe format, the root value, and the file address

If the value in the column is literal, it branches directly by literal category

def buildDecisionTree(df, root):

Identify the different charForResp

charForResp = "'"

if algorithm == 'Regression':

charForResp = ""

tmp\_root = root \* 1

df\_copy = df.copy()

Output the winning column of the decision tree, enter a list,

and output the column name of the decision column in the list

winner\_name = findDecision(df)

Determines whether the winning column is a number or a character

numericColumn = False

if dataset\_features[winner\_name] != 'object':

numericColumn = True

To ensure the integrity of the original data and prevent the data from changing,

mainly to ensure that the data of other columns besides the winning column does not change,

so as to continue the branch in the next step.

columns = df.shape[1]

for i in range(0, columns - 1):

column\_name = df.columns[i]

if df[column\_name].dtype != 'object' and column\_name != winner\_name:

df[column\_name] = df\_copy[column\_name]

Find the element in the branching column

classes = df[winner\_name].value\_counts().keys().tolist()

Traversing all classes in the branch column has two functions:

1. Display which class is currently traversed to; 2. Determine whether the current class is already leaf node

for i in range(0, len(classes)):

Find the Subdataset as in FindDecision, but discard this column of the current branch

current\_class = classes[i]

subdataset = df[df[winner\_name] == current\_class]

At the same time, the data of the first branch column is discarded and the remaining data is processed

subdataset = subdataset.drop(columns=[winner\_name])

Edit the display situation. If it is a numeric feature, the character conversion has been completed when searching for branches.

#If it is not a character feature, it is displayed with column names

if numericColumn == True:

compareTo = current\_class # current class might be <=x or >x in this case

else:

compareTo = " == '" + str(current\_class) + "'"

terminateBuilding = False

-----------------------------------------------

This determines whether it is already the last leaf node

if len(subdataset['Decision'].value\_counts().tolist()) == 1:

final\_decision = subdataset['Decision'].value\_counts().keys().tolist()[

0] # all items are equal in this case

terminateBuilding = True

At this time, only the Decision column is left, that is, all the segmentation features have been used

elif subdataset.shape[1] == 1:

get the most frequent one

final\_decision = subdataset['Decision'].value\_counts().idxmax()

terminateBuilding = True

The regression tree is judged as leaf node if the number of elements is less than 5

#elif algorithm == 'Regression' and subdataset.shape[0] < 5: # pruning condition

Another criterion is to take the standard deviation as the criterion and the sample mean in the node as the value of the node

elif algorithm == 'Regression' and subdataset['Decision'].std(ddof=0)/global\_stdev < 0.4:

get average

final\_decision = subdataset['Decision'].mean()

terminateBuilding = True

-----------------------------------------------

Here we begin to output the branching results of the decision tree.。

print(formatRule(root), "if ", winner\_name, compareTo, ":")

-----------------------------------------------

check decision is made

if terminateBuilding == True:

print(formatRule(root + 1), "return ", charForResp + str(final\_decision) + charForResp)

else: # decision is not made, continue to create branch and leafs

The size of the indent at display represented by root

root = root + 1

Call this function again for the next loop

buildDecisionTree(subdataset, root)

root = tmp\_root \* 1

Execute the code

Code:

call the function

buildDecisionTree(df, root)

Output:


Regression tree result


CART tree result