Logistic Regression

Modelling the probability of a binary outcome using the sigmoid function — fitting by maximum likelihood or gradient descent.

Logistic regression — sigmoid squashes linear output to probability

σ(wx+b) = σ(1.5x+0.0)

w=1.5

b=0.0

Definition

Logistic regression predicts the probability that a binary outcome ( $y \in \{0, 1\}$ ) is 1, given features $\mathbf{x}$ .

The model applies the sigmoid function to a linear combination of features:

$P(y=1 \mid \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$

The sigmoid squashes any real number into $(0,1)$ , making the output interpretable as a probability.

A decision boundary is drawn where $P = 0.5$ , i.e., where $\mathbf{w}^T \mathbf{x} + b = 0$ .

Key properties

Output is always strictly between 0 and 1 — never exactly 0 or 1, no matter how extreme the input
The decision boundary is always linear (a hyperplane) in the original feature space
Log-odds (logit) are linear in the features, even though probability itself is not
Fits via maximizing likelihood — there's no closed-form solution, unlike ordinary linear regression

Common mistakes

Using MSE instead of cross-entropy as the loss: MSE with a sigmoid output produces a non-convex loss landscape, making optimization unreliable — cross-entropy is convex and the standard choice
Treating the linear decision boundary as a fixed limitation: feature engineering (polynomial terms, interactions) can let logistic regression fit curved boundaries despite the model itself being linear in its inputs

Email spam classification

Features: $x_1$ = number of exclamation marks, $x_2$ = contains "FREE" (0/1). Learned weights: $w_1 = 1.2$ , $w_2 = 2.1$ , $b = -3$ .

For an email with 2 exclamation marks and "FREE": $z = 1.2(2) + 2.1(1) - 3 = 1.5$ . $P(\text{spam}) = \sigma(1.5) \approx 0.82$ . Likely spam.

Try it

If the sigmoid outputs 0.72, what class would logistic regression predict (using threshold 0.5)? What is the log-odds?

Solution

Class 1 (since $0.72 > 0.5$ ).

Log-odds = $\log(p/(1-p)) = \log(0.72/0.28) \approx \log(2.57) \approx 0.944$ .

Logistic regression models log-odds as a linear function of features — the "logit" transformation.

Related concepts

Needs first