5. Supervised learning: Cost function and Gradient descent for Logistic Regression

In this article, the main focus is on understanding how to choose a suitable cost function for logistic regression. The cost function plays a crucial role in measuring how well a given set of parameters—typically denoted as 𝑤 and 𝑏—fits the training data.

By evaluating how well these parameters perform across the training set, the cost function gives us a method to iteratively improve them, generally using an optimization algorithm like gradient descent.

Why Squared Error Doesn't Work for Logistic Regression?

Squared error (used in linear regression) produces a non-convex cost function when used with logistic regression. A non-convex function has many local minima, which makes gradient descent unreliable.

In linear regression, the model is:

𝑓(𝑥) = 𝑤⋅𝑥 + 𝑏

And the squared error cost function is:

J(w, b) = (1/2m) * ∑_i=1m (f(x⁽ⁱ⁾) - ŷ⁽ⁱ⁾)²

This function is convex, meaning it has a nice bowl shape. So, gradient descent works well—it gradually steps toward the global minimum, no matter where it starts.

In logistic regression, the prediction is:

f(x)= ŷ = 1/1+e^{-(𝑤⋅𝑥 + 𝑏)}

This function is non-linear and always gives a value between 0 and 1 (interpreted as a probability of fraud in our case).

If we plug this nonlinear 𝑓(𝑥) into the same squared error formula:

J(w, b) = (1/2m) * ∑_i=1m (1/1+e^{-(𝑤⋅𝑥 + 𝑏)} - ŷ⁽ⁱ⁾)²

...then the resulting cost function is no longer convex.

When using squared error as a cost function in logistic regression, the shape of the cost surface becomes non-convex—this means that instead of a smooth bowl shape with a single lowest point (the global minimum), the surface has multiple dips or valleys (local minima).

Gradient descent works by moving in the direction that decreases the cost function the most. If the surface is non-convex, gradient descent can:

- Get stuck in a local minimum (a dip that's not the lowest possible point),
- Fail to reach the best parameters that result in optimal predictions.

Cost Function for Logistic Regression

Let: 𝑓(𝑥^(𝑖)) = prediction for the 𝑖^𝑡ℎ training example (i.e., probability it is fraudulent)
𝑦(𝑖) = true label (1 if fraud, 0 otherwise)
𝑚 = number of training examples

For a Single Example Logistic loss Function:

If 𝑦=1: Loss = −log⁡(𝑓(𝑥))
If 𝑦=0: Loss = −log⁡(1−𝑓(𝑥))

Loss is a number that tells you how wrong your model's prediction is for a single training example. Loss is for one example. Cost is the average loss over all examples. The goal of training is to minimize loss, and therefore minimize the cost.

Here's the graph showing both log(x) (in blue) and -log(x) (in red):

The loss function commonly used for logistic regression is binary cross-entropy (or log loss), which measures how far the predicted probability is from the actual class label (0 or 1).

Here's the graph of the loss function − log ⁡(𝑓) for 𝑦 = 1 (true label 1) in logistic regression:

As you can see, the loss decreases as the predicted probability 𝑓 gets closer to 1, and increases sharply as 𝑓 approaches 0:

The curve shows that the model's loss is minimal when it is confident and correct (predicting probabilities close to 1), and grows significantly when the model is far from the correct prediction.

Here's the graph for the loss function when 𝑦 = 0 y=0:

As the predicted probability 𝑓 → 0, the loss goes to 0 — perfect prediction. As 𝑓 → 1, the loss increases sharply — the model is confidently wrong.

This curve reflects how logistic regression penalizes incorrect high-confidence predictions when the true label is 0.

The simplified loss function for a single training example is:

Loss (𝑓_{(w,b)(x⁽ⁱ⁾)},𝑦⁽ⁱ⁾) = − [ 𝑦⁽ⁱ⁾ ⋅ log ⁡ (𝑓_{(w,b)(x⁽ⁱ⁾)}) - ( 1 − 𝑦⁽ⁱ⁾ ) ⋅ log ⁡ (1 − 𝑓_{(w,b)(x⁽ⁱ⁾)}) ]

Why This Works:

When 𝑦 = 1:
Loss = − log(𝑓) (Just like before!)

When 𝑦 = 0:
Loss = − log (1−𝑓) (Also same as before!)

So this unified expression handles both cases in one line — super useful for coding and implementing things like gradient descent!

Now, let’s say you have 𝑚 training examples. The cost function (average loss across all examples) is:

𝐽 (w,b) = - (1/m) * ∑_i=1m [L(𝑓_{(w,b)(x⁽ⁱ⁾)},𝑦⁽ⁱ⁾)]

or

𝐽 (w,b) = - (1/m) * ∑_i=1m [𝑦⁽ⁱ⁾ ⋅ log ⁡ (𝑓_{(w,b)(x⁽ⁱ⁾)}) - ( 1 − 𝑦⁽ⁱ⁾ ) ⋅ log ⁡ (1 − 𝑓_{(w,b)(x⁽ⁱ⁾)})]

Gradient Descent for Logistic Regression

In this section, we'll dive into how to implement logistic regression by optimizing its parameters using gradient descent.

To fit a logistic regression model, we aim to find values for the parameters w (weights) and b (bias) that minimize the cost function 𝐽(𝑤,𝑏). This cost function quantifies how well the model's predictions align with the actual labels in the training data.

To minimize this cost function, we apply gradient descent, an optimization algorithm that iteratively updates the model parameters in the direction that reduces the cost.

The logistic regression model:

fw,b(x)= ŷ = 1/1+e^{-(𝑤⋅𝑥 + 𝑏)}

Once the model has been trained, we can use it to assess new data—for example, a new transaction that includes features like the transaction amount, location, time, and device used. The model can then predict whether the transaction is fraudulent or legitimate by estimating the probability that the label y=1 (fraud).

Here's how gradient descent works in this context. We update the parameters w and b using the following rule:

repeat{ 𝑤𝑗 := 𝑤𝑗 − 𝛼 ​ ∂𝐽/∂𝑤𝑗 ​ 𝐽 (w,b)
𝑏 := 𝑏 − 𝛼 ​ ∂𝐽/∂𝑏 ​ 𝐽 (w,b) }

By applying the rules of calculus and working through the logistic regression cost function, this derivative ∂𝐽/∂𝑤𝑗 ​ 𝐽 (w,b) evaluates to:

∂𝐽/∂𝑤𝑗 ​ 𝐽 (w,b) = (1/m) * ∑_i=1m ​ (𝑓_{(w,b)(x⁽ⁱ⁾)} - 𝑦⁽ⁱ⁾) ​ x_j⁽ⁱ⁾
∂𝐽/∂𝑏 ​ 𝐽 (w,b) = (1/m) * ∑_i=1m ​ (𝑓_{(w,b)(x⁽ⁱ⁾)} - 𝑦⁽ⁱ⁾)

As a quick reminder: when updating the parameters 𝑤𝑗 and 𝑏, we don't update them one at a time while computing the gradients. Instead, we:
- First compute all the necessary gradient values (i.e., the right-hand side of the update rules),
- Then simultaneously update all the parameters using those values.

Let's plug the gradients we derived earlier into the gradient descent update rules. The update formulas become:

repeat{
𝑤𝑗 := 𝑤𝑗 − 𝛼 ​ [(1/m) * ∑_i=1m ​ (𝑓_{(w,b)(x⁽ⁱ⁾)} - 𝑦⁽ⁱ⁾) ​ x_j⁽ⁱ⁾]
𝑏 := 𝑏 − 𝛼 ​ [(1/m) * ∑_i=1m ​ (𝑓_{(w,b)(x⁽ⁱ⁾)} - 𝑦⁽ⁱ⁾)]
}

These updates form the core of gradient descent for logistic regression.

These equations look really similar to the ones we used for linear regression. Are logistic regression and linear regression actually the same?

The answer is no. While the gradient descent update equations look similar, the fundamental difference lies in the function 𝑓 (𝑥):
- In linear regression, 𝑓(𝑥) = 𝑤 ​ 𝑥 + 𝑏
- In logistic regression, 𝑓(𝑥) = 1/1+e^{-(𝑤⋅𝑥 + 𝑏)}


The sigmoid function squashes the output to lie between 0 and 1, turning it into a probability estimate—perfect for classification tasks like fraud detection. That change makes a huge difference in the behavior and purpose of the model.

So while the mechanics of gradient descent might feel familiar, logistic regression is fundamentally different from linear regression.

In linear regression, we talked about how to monitor gradient descent to ensure it converges—i.e., that the cost function is decreasing with each iteration. The same principle applies here. You can:

- Track the value of the cost function 𝐽(𝑤,𝑏) over iterations
- Plot a learning curve
- Stop training once the cost function stabilizes or drops below a desired threshold

This helps confirm that your model is learning correctly.

Another trick we discussed during linear regression is feature scaling—and yes, it's just as valuable here. Feature scaling ensures that all features (e.g., transaction amount, transaction time) lie within a similar range, typically between -1 and 1.

This helps gradient descent converge faster and more reliably, as the optimization landscape becomes smoother and easier to navigate.