Logistic Regression: Cost Function
To train the parameters 𝑤 and 𝑏, we need to define a cost function.
Recap:
𝑦̂(𝑖) = 𝜎(𝑤𝑇𝑥(𝑖) + 𝑏), where 𝜎(𝑧(𝑖))=
1
1+ 𝑒−𝑧(𝑖)
𝑥(𝑖) the i-th training example
𝐺𝑖𝑣𝑒𝑛 {(𝑥(1), 𝑦(1) ), ⋯ , (𝑥(𝑚), 𝑦(𝑚) )}, 𝑤𝑒 𝑤𝑎𝑛𝑡 𝑦̂(𝑖) ≈ 𝑦(𝑖)
Publicité
Loss (error) function:
The loss function measures the discrepancy between the prediction (𝑦̂(𝑖)) and the desired output (𝑦(𝑖)).
In other words, the loss function computes the error for a single training example.
𝐿(𝑦̂(𝑖), 𝑦(𝑖)) =
(𝑦̂(𝑖) − 𝑦(𝑖))2
1
2
𝐿(𝑦̂(𝑖), 𝑦(𝑖)) = −( 𝑦(𝑖) log(𝑦̂(𝑖)) + (1 − 𝑦(𝑖))log (1 − 𝑦̂(𝑖))
Publicité
•
•
If 𝑦(𝑖) = 1: 𝐿(𝑦̂(𝑖), 𝑦(𝑖)) = − log(𝑦̂(𝑖)) where log(𝑦̂(𝑖)) and 𝑦̂(𝑖) should be close to 1
If 𝑦(𝑖) = 0: 𝐿(𝑦̂(𝑖), 𝑦(𝑖)) = − log(1 − 𝑦̂(𝑖)) where log(1 − 𝑦̂(𝑖)) and 𝑦̂(𝑖) should be close to 0
Cost function
The cost function is the average of the loss function of the entire training set. We are going to find the
parameters 𝑤 𝑎𝑛𝑑 𝑏 that minimize the overall cost function.
𝐽(𝑤, 𝑏) =
Publicité
1
𝑚
𝑚
∑ 𝐿(𝑦̂(𝑖), 𝑦(𝑖))
𝑖=1
= −
1
𝑚
Publicité
𝑚
∑[( 𝑦(𝑖) log(𝑦̂(𝑖)) + (1 − 𝑦(𝑖))log (1 − 𝑦̂(𝑖))]
𝑖=1