7 concepts25 worked examples33 exercises5 exam-level7 figures
What are you here for?
07 Generalized linear models: logistic regression, Newton-Raphson and IRLS, more than two classes, and Poisson regression
Start with this
One question before you read anything. Getting it wrong is the point: it shows you what this section is for.
§07.1 — is a least squares fit to 0/1 labels a probability?
In the least squares section, six students who studied $x=1,\dots,6$ hours were coded pass $=1$ and fail $=0$, and the fitted line was $\hat y=-0.2+0.2x$. A classmate says the line's height is each student's chance of passing.
Find(a) True or false: for every $x$ from $0$ to $8$ hours, $\hat y$ is a probability of passing.
Given
$x=1,2,3,4,5,6$ hours; labels $y=0,0,1,0,1,1$
fitted line $\hat y=-0.2+0.2x$
Hint 1/4
A probability has to stay in $[0,1]$. Test the claim where a straight line is most extreme: at the two ends of the range.
Hint 2/4
Evaluate $\hat y=-0.2+0.2x$ at $x=0$ and at $x=8$.
Hint 3/4
The line is $\hat y=-0.2+0.2x$ on $0\le x\le 8$. At $x=0$: $-0.2$. At $x=8$: $-0.2+1.6=1.4$.
Hint 4/4
Both values leave $[0,1]$, so the claim is false.
Show solution
A line is monotone, so its extremes on an interval sit at the endpoints; two evaluations settle the claim.
Evaluate at the endpoints
$$\hat y(0)=-0.2+0.2\cdot0=-0.2$$
The smallest value on the interval, since the slope is positive.
$$\hat y(8)=-0.2+0.2\cdot8=1.4$$
The largest value on the interval.
Answer $$\boxed{\text{False: }\hat y\text{ runs from }-0.2\text{ to }1.4}$$
Check
Solve $0\le-0.2+0.2x\le1$ instead: it holds only for $1\le x\le6$, so every input outside the six students' range breaks the claim.
Any line with a nonzero slope leaves $[0,1]$ somewhere, so a probability model needs a curve with a floor and a ceiling.
Six students studied 1 to 6 hours, and three of them passed. A straight line fitted to their pass and fail marks says that a student who studies 8 hours passes with 'probability' $1.4$, and one who studies 0 hours with $-0.2$. A different curve fitted to the same six marks says $0.996$ and $0.014$.
By the end you can fit that curve yourself: write its likelihood, take Newton steps by hand, add a penalty, extend it to more than two classes, and fit counts with the same recipe.
In 60 seconds
Logistic regression passes a linear score through the to get $\Pr(Y=1\mid x)$, is fitted by Newton-Raphson on the Bernoulli log-likelihood, which amounts to iteratively reweighted least squares, and together with linear and Poisson regression forms the family of generalized linear models.
Reading $e^{\beta_1}$ as a change in probability: it multiplies the odds, and how much $\pi$ moves depends on where it starts.
Weighting by $\pi_i$ or subtracting the Newton step: the weights are $\pi_i(1-\pi_i)$, and for a maximum the update adds $(X^TWX)^{-1}X^T(y-\pi)$.
Taking the log of the counts: Poisson regression models $\log E[Y\mid x]$, the log of the mean, so hours with $0$ events stay in the fit.
Two course documents give different weights:
Chapter 1 slides, undergraduate line: midterm 25, final 25, four quizzes 20, two-phase project 30.
STARS syllabus page printed on 21 September 2026: midterm 30, final 30, problem sets and quizzes 20, project 20. It is the later document; confirm which split applies.
The STARS weekly list puts generalized linear models fifth; the lecture slides make them chapter 7.
How much time do you have?
10 minutes
The model and how to read it, one Newton step done by hand, and the count model, each with a worked example.
The 60-second card · The logistic model · Newton-Raphson and iteratively reweighted least squares · Poisson regression: counts with a log link · Formula card
45 minutes
Every result once, then the Newton step from a full solution down to a bare problem.
The 60-second card · The logistic model · Fitting by maximum likelihood · Newton-Raphson and iteratively reweighted least squares · Penalized logistic regression · More than two classes · Generalized linear models · Poisson regression: counts with a log link · Scaffolding comes off · Formula card
full read
Adds the derivations, the look-alike pairs and mixed practice where you decide the method yourself.
The opening pages · Recall first · The logistic model · Fitting by maximum likelihood · Newton-Raphson and iteratively reweighted least squares · Penalized logistic regression · More than two classes · Generalized linear models · Poisson regression: counts with a log link · Look-alike pairs · Method boxes · Scaffolding comes off · Full exam-style question · Practice set · Check yourself
By the end of this section
Compute probabilities, odds and log-odds from logistic coefficients, read a coefficient as an , and turn a probability into a decision with a threshold.
Derive the logistic log-likelihood and its gradient $X^T(y-\pi)$, and solve the score equations in closed form when the only predictor is binary.
Perform Newton-Raphson steps by hand, for one coefficient and for two, and write the logistic update as iteratively reweighted least squares.
Explain what $\ell_1$ and $\ell_2$ penalties do to a logistic fit as $\lambda$ runs from $0$ to $\infty$, including data that a line separates.
Compute class probabilities and decision boundaries in the multinomial model with a baseline class, and count its parameters.
Write the Gaussian, Bernoulli, binomial and Poisson distributions in exponential-family form and name the link of linear, logistic and Poisson regression.
Fit and interpret a Poisson regression: expected counts, , the score equations $X^Ty=X^T\hat\mu$ and a Newton step.
Syllabus coverage
covered
Generalized linear models — covered
, , the functions a, b, c
linear predictor, mean response and
linear, logistic and Poisson regression as three generalized linear models fitted by Newton-Raphson
The lecture's chapter title and the STARS weekly line are the same words. The STARS list puts this topic fifth; the lecture slides make it chapter 7.
covered
The classification problem — covered
A categorical response
coding classes as numbers
why three or more unordered classes cannot be coded
why two classes can
The chapter opens here and returns to the 0/1 coding of the least squares section, which becomes this section's first failure.
covered
Logistic regression — covered
Logistic function, odds and log-odds
predictions and
a categorical predictor
several predictors and
Simple and multiple logistic regression are written in one vector form.
covered
Maximum likelihood for logistic regression — covered
Bernoulli likelihood, log-likelihood, gradient and score equations
no closed form in general
concavity of the log-likelihood
The lecture notes pose the concavity question; the answer is proved here from the .
covered
Newton-Raphson and iteratively reweighted least squares — covered
Newton's update in one and several variables, a , as the cheaper alternative, and the logistic update as weighted least squares on an .
Gradient descent comes from the chapter's lecture notes.
covered
Regularized logistic regression — covered
l1 and l2 penalties on the log-likelihood, the unpenalized intercept, and the limits of small and large lambda.
The same penalties as ridge and the lasso, attached to a log-likelihood instead of a sum of squares.
covered
Logistic regression for more than two classes — covered
K minus 1 log-odds against a baseline class, the class probabilities, the parameter count and fitting by Newton's method.
The lecture notes call the response distribution multinoulli.
covered
Poisson regression — covered
Counts with a log link
the log-likelihood
the score equations
rate ratios and a comparison of observed and expected counts
The lecture illustrates it with a classic count table; the data here are new.
off syllabus
Separable data — off syllabus
When a line separates the two classes, the log-likelihood keeps rising and the maximum likelihood estimate does not exist; a penalty restores a finite answer.
Further reading: not on the slides. Two points are enough to show it, and it explains why software warns about fitted probabilities of exactly 0 or 1.
off syllabus
Closed-form fits with one binary predictor — off syllabus
Solving the score equations by hand when the only predictor is 0 or 1, for logistic and for Poisson regression.
Further reading: the slides fit a binary predictor numerically. Solving by hand shows that the fit reproduces the two group rates, which gives every Newton computation an independent check.
off syllabus
Mean of an exponential family — off syllabus
The identity E[Y] = b'(theta), used only as a check on the exponential-family forms.
Further reading: the slides state the family and the links. No exercise depends on this identity.
Recall first
Bernoulli and Poisson distributions
Bernoulli: $\Pr(Y=y)=p^y(1-p)^{1-y}$ for $y\in\{0,1\}$, mean $p$. Poisson: $\Pr(Y=y)=\frac{\mu^y}{y!}e^{-\mu}$ for $y=0,1,2,\dots$, mean $\mu$ and variance $\mu$.
They are the response distributions of logistic and Poisson regression.
Maximum likelihood
$\hat\theta^{\mathrm{MLE}}=\arg\max_\theta p(D\mid\theta)=\arg\max_\theta\log p(D\mid\theta)$; a log or a positive constant factor does not move a maximizer. For $n$ independent Bernoulli trials with $N_1$ ones, $\hat p=N_1/n$.
Every model in this section is fitted by maximizing a log-likelihood.
Least squares and the 0.5 cut
With design matrix $X$ (rows $x_i^T=[1,x_{i1},\dots,x_{ip}]$), $\mathrm{RSS}(\beta)=(y-X\beta)^T(y-X\beta)$ has gradient $-2X^T(y-X\beta)$ and minimizer $(X^TX)^{-1}X^Ty$. With 0/1 labels, predicting class 1 where the fitted value exceeds $0.5$ gives a classifier.
The section's first failure starts from that classifier, and each Newton step turns out to be a weighted version of this formula.
Ridge and lasso penalties
Ridge adds $\lambda\sum_j\beta_j^2$ to the RSS and the lasso adds $\lambda\sum_j\lvert\beta_j\rvert$; the lasso can set coefficients exactly to $0$. Read as MAP estimates, they come from Gaussian and Laplace priors.
The same two penalties are attached to the logistic log-likelihood.
First-order Taylor expansion
For small $\Delta$: $f'(w+\Delta)\approx f'(w)+\Delta f''(w)$. In several variables $\nabla f(w+\Delta)\approx\nabla f(w)+H(w)\Delta$, where the Hessian $H$ holds the second partial derivatives.
Newton-Raphson is this approximation set to zero and solved for the step.
2 × 2 inverse
$\begin{pmatrix}a&b\\c&d\end{pmatrix}^{-1}=\frac{1}{ad-bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}$ when $ad-bc\neq0$.
Every two-coefficient Newton step on this page is solved with it.
Try it yourself first (2 questions)
1§07.2 — the maximum likelihood estimate of a failure rate
A plant inspects 10 welds and 2 of them fail. Treat the outcomes as independent Bernoulli trials with failure probability $p$; the log-likelihood is $l(p)=2\log p+8\log(1-p)$.
Find(a) Which $p$ maximizes $l(p)$?
Given
$n=10$ welds, $2$ failures
$l(p)=2\log p+8\log(1-p)$
Hint 1/4
The maximizer of a smooth function inside $(0,1)$ has zero derivative: find where $l'(p)=0$.
Hint 2/4
$\frac{d}{dp}\log p=\frac1p$ and $\frac{d}{dp}\log(1-p)=-\frac{1}{1-p}$.
Hint 3/4
With $l(p)=2\log p+8\log(1-p)$: $l'(p)=\frac2p-\frac{8}{1-p}$. Set it to $0$.
Hint 4/4
$2(1-p)=8p$, so $p=0.2$: the observed failure proportion.
Show solution
One parameter and a smooth log-likelihood: setting the derivative to zero is the shortest route.
$X$ is $n\times(p+1)$ with rows $x_i^T$; $\pi$ stacks $\pi(x_i;\beta)$
$\pi$ is recomputed at every Newton step.
$W=\mathrm{diag}\big(\pi_i(1-\pi_i)\big)$
the weight matrix
diagonal, one weight per point, largest when $\pi_i=0.5$
Evaluated at the old coefficients.
$z=X\beta^{\mathrm{old}}+W^{-1}(y-\pi)$
the adjusted response
the target of the weighted least squares fit in one IRLS step
Changes every step.
$\nabla f,\ H$
gradient, Hessian
column of first partial derivatives; matrix of second partial derivatives
For $l(\beta)$: $\nabla l=X^T(y-\pi)$, $H=-X^TWX$.
$w_t,\ \gamma$
the iterate at step t, the step size
Newton's sequence $w_0,w_1,\dots$; a damping factor $\gamma\in(0,1]$
The lecture notes index iterates by $n$.
$\lambda$
lambda
penalty weight in regularized logistic regression
The intercept is never penalized.
$K,\ \beta_j,\ \theta$
number of classes, class-j coefficients, all of them
$\beta_j$ for $j=1,\dots,K-1$; class $K$ is the baseline; $\theta=\{\beta_1,\dots,\beta_{K-1}\}$
$\theta$ also names the canonical parameter of a GLM.
$\theta,\ \phi,\ a,\ b,\ c$
canonical parameter, dispersion, the three functions
the parts of $p(y\mid\theta,\phi)=\exp\big(\frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi)\big)$
For Bernoulli and Poisson, $a(\phi)=1$.
$\eta=\beta^Tx,\ \mu=E[Y\mid x],\ g$
linear predictor, mean response, link
the GLM's three pieces, tied by $g(\mu)=\eta$
The link acts on the mean, not on the data.
Conventions used here
Natural logarithm.
$\log$ always means $\ln$. Odds ratios such as $e^{0.7}\approx2.014$ use base $e$.
Every derivative on this page assumes base e.
Which class is 1.
$\pi(x;\beta)$ is the probability of the class coded $1$. Recoding the two classes the other way flips the sign of every coefficient and changes no prediction.
With two classes the coding is harmless; the order problem starts at three.
Gradients are columns.
Following the chapter's lecture notes, $\nabla l$ is a column vector. The least squares section wrote derivatives as rows; transposing gives the same equations.
Newton's update multiplies the gradient by a matrix, which needs a column.
Iterations are indexed by t.
Newton iterates are $w_0,w_1,w_2,\dots$ or $\beta^{(0)},\beta^{(1)},\dots$; the lecture notes write $w_n$. Here $n$ is kept for the number of data points.
One letter for two things causes wrong sums.
Newton finds stationary points.
The update $w_{t+1}=w_t-H^{-1}\nabla f$ is the same for a minimum and a maximum. Because $l(\beta)$ is concave, the stationary point it reaches is the maximum.
No sign change is needed as long as the Hessian of the function itself is used.
Decision rules.
A threshold $t$ predicts class $1$ when $\pi(x)\ge t$. The lecture's risk-neutral rule uses $t=0.5$ and its risk-averse example $t=0.1$.
A tie at exactly t goes to class 1.
The intercept is free.
In regularized logistic regression the penalty runs over $j=1,\dots,p$; $\beta_0$ is never penalized.
Penalizing it would pull every probability toward 0.5 whatever the class balance.
The baseline class.
In the $K$-class model the baseline is the last class, $K$. Any class could serve; the fitted probabilities do not depend on the choice.
Only the coefficients change meaning when the baseline changes.
When the MLE does not exist.
If the log-likelihood keeps increasing without reaching a maximum, we say the maximum likelihood estimate does not exist; we do not write $\hat\beta=\infty$.
This happens exactly when a line separates the classes.
Rounding.
Intermediate steps keep at least four significant figures; final answers are rounded to three decimals unless an exact fraction is asked for. Newton iterates keep four decimals so that the convergence stays visible, and hours, counts of hours and amounts in TL are given to one decimal.
Newton's later steps change only the third or fourth decimal.
7.1The logistic model: a linear score turned into a probability
Turns a linear score into a probability between 0 and 1; the log-odds, not the probability, is linear in the inputs.
The least squares section already classified with a line through 0/1 labels. The line put four of the six students on the correct side of $0.5$, but its height is not a chance of anything.
Solvable with what we have
Fit a line to 0/1 labels and classify by the side of $0.5$: the six students get the boundary $x=3.5$ hours.
Rank students: more hours always means a higher fitted value.
Code two classes as $0$ and $1$ either way round; the predicted classes do not change.
Not solvable yet
Say how likely a student with 8 hours is to pass: the line answers $1.4$.
Give every input a probability that is guaranteed to stay in $[0,1]$.
Code three unordered illnesses as $1,2,3$ without inventing an order and an equal spacing.
Read the least squares line $\textcolor{#8250df}{\hat y=-0.2+0.2x}$ as a probability. At 8 hours it gives $1.4$, at 0 hours $-0.2$. Clipping to $[0,1]$ turns both into certainty, 1 and 0, from six students. The curve built in this block gives $\textcolor{#1f6feb}{0.996}$ and $\textcolor{#1f6feb}{0.014}$.
Why it fails
A straight line has no floor and no ceiling, and least squares treats labels as positions on a ruler. A probability needs a curve squeezed into $(0,1)$.
DefinitionDefinition 7.1: Logistic regression
Conditions
$Y\in\{0,1\}$ and $X=[1,X_1,\dots,X_p]^T$, with a leading $1$ for the intercept
$\beta=[\beta_0,\beta_1,\dots,\beta_p]^T$; with one input, $\beta^TX=\beta_0+\beta_1X$
the model describes $Y$ given $X$; it says nothing about how $X$ itself is distributed
Build the same linear score as in linear regression, then pass it through the logistic function, which sends any real number into $(0,1)$. Read backwards, the log-odds of class 1 is linear in the inputs: that is the sense in which this model is linear.
Proof
$1-\pi=\frac{1+e^{\beta^TX}-e^{\beta^TX}}{1+e^{\beta^TX}}=\frac{1}{1+e^{\beta^TX}}$, so $\frac{\pi}{1-\pi}=e^{\beta^TX}$, and the log of that is $\beta^TX$.
Limits with one input and $\beta_1>0$: as $X\to-\infty$ the score goes to $-\infty$ and $\pi\to0$; as $X\to+\infty$, $\pi\to1$. With $\beta_1<0$ the two limits swap.
Dividing top and bottom by $e^{\beta^TX}$ gives the equivalent form $\pi=\frac{1}{1+e^{-\beta^TX}}$, often quicker on a calculator.
The six students of the least squares section: fail as hollow circles, pass as filled ones. The line $\textcolor{#8250df}{\hat y=-0.2+0.2x}$ leaves the shaded band at both ends; the logistic curve $\textcolor{#1f6feb}{\pi(x)}$, fitted to the same marks, flattens toward $0$ and $1$ instead. Both cross $0.5$ at $x=3.5$ hours.
Looks like this, but is not
The model $\pi(x)=0.1+0.05x$ is also linear in $x$ and gives numbers that look like probabilities.
It is linear in the probability itself, so it reaches $1$ at $x=18$ and $0$ at $x=-2$, and beyond those it leaves $[0,1]$. The logistic model is linear in the log-odds, and $\pi$ stays inside $(0,1)$ for every $x$.
probability π
odds π/(1 − π)
log-odds
$0.1$
$0.111$
$-2.197$
$0.2$
$0.25$
$-1.386$
$0.5$
$1$
$0$
$0.8$
$4$
$1.386$
$0.9$
$9$
$2.197$
Log-odds are symmetric around $\pi=0.5$: swapping $\pi$ and $1-\pi$ flips the sign. Equal steps in log-odds are large steps in $\pi$ near $0.5$ and small steps near $0$ or $1$.
Pass probabilities at 0 and 8 hours from the fitted curve
The logistic fit to the six students is $\hat\beta_0=-4.249$, $\hat\beta_1=1.214$; we compute it ourselves in the Newton block. Find $\pi(0)$, $\pi(8)$ and the number of hours where $\pi=0.5$.
Find$\pi(0)$, $\pi(8)$ and the $x$ with $\pi(x)=0.5$.
Given
$\hat\beta_0=-4.249$, $\hat\beta_1=1.214$ per hour
The data are symmetric: replacing $x$ by $7-x$ and swapping pass and fail gives the same six marks, so the fit must satisfy $\pi(7-x)=1-\pi(x)$. Indeed $\pi(7)=1/(1+e^{-4.249})=0.986=1-0.014$.
Same boundary as the least squares line, 3.5 hours, but now the heights are probabilities: this closes the gap the section opened with.
Pump failure from vibration: three probabilities and two decision rules
A plant models the chance that a pump fails within a month from its vibration level $x$ in mm/s: $\operatorname{logit}\pi(x)=-6+1.5x$. Find $\pi$ at $x=2,4,5$. Then find the vibration from which each rule flags a pump: risk-neutral, flag if $\pi\ge0.5$, and risk-averse, flag if $\pi\ge0.1$.
FindThree probabilities and two flagging levels on $x$.
Given
$\operatorname{logit}\pi(x)=-6+1.5x$, $x$ in mm/s
flag if $\pi(x)\ge t$, with $t=0.5$ or $t=0.1$
Solution
Compare scores, not probabilities: $\pi\ge t$ is the same as $\eta\ge\operatorname{logit}(t)$, which is linear in $x$ and solves in one line.
Solve for $x$; the slope is positive, so the inequality keeps its direction.
Answer $$\boxed{\begin{aligned}&\pi(2)=0.047,\quad \pi(4)=0.5,\quad \pi(5)=0.818\\ &\text{flag from }x=4\ (t=0.5)\ \text{or from }x=2.535\ (t=0.1)\end{aligned}}$$
Check
Plug back: $\eta(2.535)=-6+3.803=-2.197$, and $1/(1+e^{2.197})=0.100$, the risk-averse threshold exactly.
The risk-averse rule also flags every pump between $2.535$ and $4$ mm/s, which the risk-neutral rule lets run.
A lower threshold moves the flagging point down the input scale; the fitted curve itself does not change.
What one more hour of study does to the odds and to the probability
For the six students, $\hat\beta_0=-4.249$ and $\hat\beta_1=1.214$ per hour. By what factor do the odds of passing grow per extra hour? How much does the probability grow from 3 to 4 hours, and from 6 to 7 hours?
FindThe odds ratio per hour and two changes in probability.
Given$\hat\beta_0=-4.249$, $\hat\beta_1=1.214$
Solution
Work on the odds scale first, where one hour always means the same multiplication, then convert to probabilities to see the uneven effect.
The same odds factor now moves the probability about a tenth as much.
Answer $$\boxed{\begin{aligned}&\text{odds}\times3.367\text{ per hour}\\ &\Delta\pi=0.295\ (3\to4),\qquad \Delta\pi=0.032\ (6\to7)\end{aligned}}$$
Check
Check on the odds directly: $\text{odds}(3)=0.3527/0.6473=0.5449$ and $\text{odds}(4)=0.6473/0.3527=1.8353$, a ratio of $3.368$, which is $e^{1.214}=3.367$ up to rounding.
A coefficient is a statement about odds; to talk about probabilities you must say where on the curve you start.
Outdoor sensors fail more often, yet their coefficient is negative
A fitted model for the chance that a sensor unit fails within a year uses its operating temperature $T$ in °C and $O=1$ for outdoor units: $\operatorname{logit}\pi=-6+0.1T-0.8O$. Outdoor units typically run at 45 °C, indoor ones at 30 °C. Compare the two kinds at the same temperature, 40 °C, and at their typical temperatures.
FindFailure probabilities at equal and at typical temperatures, and why they point in opposite directions.
Given
$\operatorname{logit}\pi=-6+0.1T-0.8O$
typical temperatures: indoor $30$ °C, outdoor $45$ °C
Solution
Hold temperature fixed first, which is what a coefficient in a multiple model describes, then let it vary the way it does in the plant.
Recompute the typical-temperature gap from the probabilities alone, kept to four decimals: $\operatorname{logit}(0.0911)-\operatorname{logit}(0.0474)=-2.300-(-3.001)=0.70$, the same split.
A coefficient in a multiple model compares units that agree on every other input; a comparison of groups mixes in whatever else differs between them.
Checkpoint
§07.1 — probability from a score of zero
A logistic model for whether a delivery arrives late uses its distance $x$ in km: $\operatorname{logit}\pi(x)=-3+0.5x$. A dispatcher wants the chance that a 6 km delivery is late.
Find(a) What is $\pi(6)$?
Given
$\operatorname{logit}\pi(x)=-3+0.5x$
$x=6$ km
Hint 1/4
Find the score at $x=6$ first; the probability is a function of the score.
Hint 2/4
$\pi=\frac{1}{1+e^{-\eta}}$ with $\eta=-3+0.5x$.
Hint 3/4
At $x=6$: $\eta=-3+0.5\cdot6=0$.
Hint 4/4
$\pi(6)=\frac{1}{1+e^{0}}=0.5$.
Show solution
Score first, then the logistic function; with a zero score no calculator is needed.
Score
$$\eta(6)=-3+0.5\cdot6=0$$
The linear part of the model.
Probability
$$\pi(6)=\frac{1}{1+e^{-0}}=\frac12$$
$e^0=1$.
Answer $$\boxed{\pi(6)=0.5}$$
Check
Odds route: $\text{odds}=e^{0}=1$, and $1/(1+1)=0.5$.
Where the score is $0$, the model is undecided; that input is the $0.5$ boundary.
⚠ Reading the score as a probability
the linear score is the first number computed, and it looks like an answer
wrong$$\pi(6)=-3+0.5\cdot6=0$$
right$$\pi(6)=\frac{1}{1+e^{-0}}=0.5$$
⚠ Treating a coefficient as a change in probability
in linear regression a slope adds to the prediction, and the habit carries over
7.2Fitting by maximum likelihood: the Bernoulli log-likelihood and its score
Chooses the coefficients that make the observed labels most probable; the gradient $X^T(y-\pi)$ must vanish, in general only numerically.
The curve above needs numbers for $\beta$. Least squares will not supply them well: the model is not linear in $\beta$ and the labels are not Gaussian. The Bernoulli likelihood is the natural fit.
TheoremResult 7.2: The logistic log-likelihood and its gradient
Conditions
$Y_1,\dots,Y_n$ independent given the inputs, $Y_i\sim\mathrm{Bernoulli}\big(\pi(x_i;\beta)\big)$
rows $x_i=[1,x_{i1},\dots,x_{ip}]^T$ stacked in $X$; the vector $\pi$ stacks $\pi(x_i;\beta)$
Each point adds the log of the probability the model gave to its actual label. The gradient adds up the residuals $y_i-\pi_i$, each times its input vector; at the maximum these weighted residuals cancel. The log-likelihood is concave, so that point is the global maximum, but solving $X^T(y-\pi)=0$ has no closed form in general.
Proof
Likelihood, by independence: $L(\beta)=\prod_{i:y_i=1}\pi_i\prod_{i:y_i=0}(1-\pi_i)=\prod_i\pi_i^{y_i}(1-\pi_i)^{1-y_i}$.
With $\eta_i=\beta^Tx_i$: $\log\pi_i=\eta_i-\log(1+e^{\eta_i})$ and $\log(1-\pi_i)=-\log(1+e^{\eta_i})$. Substituting leaves $y_i\eta_i-\log(1+e^{\eta_i})$ for each point.
Gradient: $\frac{\partial}{\partial\beta}\log(1+e^{\eta_i})=\frac{e^{\eta_i}}{1+e^{\eta_i}}x_i=\pi_ix_i$ and $\frac{\partial}{\partial\beta}y_i\eta_i=y_ix_i$.
Concavity: once more, $\frac{\partial\pi_i}{\partial\beta}=\pi_i(1-\pi_i)x_i$, so the Hessian is $-\sum_i\pi_i(1-\pi_i)x_ix_i^T$. For any vector $v$ this gives $v^T\nabla^2l\,v=-\sum_i\pi_i(1-\pi_i)(x_i^Tv)^2\le0$: $l$ is concave, and every stationary point is a global maximum.
One point's term in $l(\beta)$ as a function of its score $\eta=\beta^Tx$. A label 1 adds $\textcolor{#1f6feb}{\log\pi}$, which rises toward $0$ as the score grows; a label 0 adds $\textcolor{#8250df}{\log(1-\pi)}$, the mirror image. Both bend downward everywhere, so their sum over the data, the log-likelihood, is concave.
Looks like this, but is not
Minimizing $\sum_i\big(y_i-\pi(x_i;\beta)\big)^2$, least squares on the probabilities, sounds like the natural fit.
It is a different estimator. It does not come from the Bernoulli model, and in general it is not convex in $\beta$, so a minimum found by search need not be the global one. The course fits logistic regression by maximum likelihood.
The six students' log-likelihood at zero and at the fit
Evaluate $l(\beta)$ for the six students at $\beta=(0,0)$ and at the fit $\hat\beta=(-4.249,\,1.214)$. Which students cost the fit the most?
Find$l(0,0)$, $l(\hat\beta)$ and the largest per-point costs.
Given
$x=1,2,3,4,5,6$; $y=0,0,1,0,1,1$
at the fit: $\hat\pi=(0.0459,\, \allowbreak 0.1393,\, \allowbreak 0.3527,\, \allowbreak 0.6473,\, \allowbreak 0.8607,\, \allowbreak 0.9541)$
Solution
Use the per-point form $y_i\log\pi_i+(1-y_i)\log(1-\pi_i)$: with the probabilities known it is one log per student.
At zero
$$\pi_i=\tfrac12\ \text{for all }i\ \Rightarrow\ l(0,0)=6\log\tfrac12=-4.159$$
A zero score gives every student probability one half, whatever the label.
Compare likelihoods instead of logs: $e^{-2.478}/e^{-4.159}=e^{1.681}\approx5.4$, so the fit makes the six observed labels about 5.4 times as probable as a coin flip does.
The log-likelihood is a sum of per-point costs, and the points that go against the trend dominate it.
Night-shift welds: one binary predictor gives the fit in closed form
Further reading. A plant inspects 400 day-shift welds, 20 of which fail, and 300 night-shift welds, 30 of which fail. Model the failure probability by $\operatorname{logit}\pi=\beta_0+\beta_1x$ with $x=1$ for night. Solve the score equations by hand.
Find$\hat\beta_0$, $\hat\beta_1$ and the odds ratio of night against day.
Given
day, $x=0$: $20$ failures in $400$ welds
night, $x=1$: $30$ failures in $300$ welds
Solution
Every weld in a group shares the same $\pi$, so each score equation becomes a statement about a group total; that is what makes a closed form possible here.
Try a point that is not the fit, $\beta=(0,0)$: every $\pi_i=0.5$, so $\sum_i\pi_i=3$ still matches but $\sum_ix_i\pi_i=10.5\neq14$. The gradient there is $(0,\,3.5)$, and the fit must move.
At the maximum the fitted probabilities reproduce the observed totals along every column of $X$; a quick column check catches a wrong fit.
Checkpoint
§07.2 — one point's term in the log-likelihood
A spam filter's logistic model gives an email the score $\eta_i=\beta^Tx_i$. The email turns out not to be spam, so its label is $y_i=0$.
Find(a) Which term does this email add to $l(\beta)$?
Given
label $y_i=0$
score $\eta_i=\beta^Tx_i$
Hint 1/4
A point with label 0 contributes the log of the probability the model gave to label 0.
Hint 2/4
That is $\log(1-\pi_i)$, and $1-\pi_i=\frac{1}{1+e^{\eta_i}}$.
Hint 3/4
Here $y_i=0$, so in $y_i\log\pi_i+(1-y_i)\log(1-\pi_i)$ only the second part survives.
Hint 4/4
The term is $\log\frac{1}{1+e^{\eta_i}}=-\log(1+e^{\eta_i})$.
Show solution
Start from the general per-point term, then set $y_i=0$.
for a loss one writes predicted minus observed, and the habit carries over to a log-likelihood
wrong$$\nabla l(\beta)=X^T(\pi-y)$$
right$$\nabla l(\beta)=X^T(y-\pi)$$
7.3Newton-Raphson and iteratively reweighted least squares
Solves the score equations with tangent steps; for logistic regression each step is a weighted least squares fit.
The score equations $X^T(y-\pi)=0$ cannot be solved by algebra, so we solve them the way a calculator finds a square root: guess, linearize, correct, repeat.
MethodMethod 7.3: Newton-Raphson, and its logistic form (IRLS)
Conditions
$f$ twice differentiable, with the Hessian $H$ invertible at each iterate
logistic regression: $\pi$ and $W=\mathrm{diag}\big(\pi_i(1-\pi_i)\big)$ are computed at $\beta^{\mathrm{old}}$
start at $\beta=0$, where every $\pi_i=0.5$, and stop when the change falls below a tolerance
Replace the gradient by its tangent at the current point and jump to where that tangent is zero. For logistic regression the Hessian is $-X^TWX$, so the jump is a weighted least squares fit to an adjusted response $z$. The weights $\pi_i(1-\pi_i)$ change every step, hence iteratively reweighted.
Proof
One variable: $0=f'(w^\ast)\approx f'(w_t)+(w^\ast-w_t)f''(w_t)$, so the root is near $w_t-f'(w_t)/f''(w_t)$. That point is the next iterate.
Several variables: the same expansion with $\nabla f$ and $H$ gives $w_{t+1}=w_t-H^{-1}\nabla f(w_t)$. A step size $\gamma\in(0,1)$ gives the damped version $w_t-\gamma H^{-1}\nabla f(w_t)$.
For $f=l$: $\nabla l=X^T(y-\pi)$ and $H=-X^TWX$, because $\partial\pi_i/\partial\beta=\pi_i(1-\pi_i)x_i$. So $\beta^{\mathrm{new}}=\beta^{\mathrm{old}}+(X^TWX)^{-1}X^T(y-\pi)$.
Factor out $(X^TWX)^{-1}X^TW$: write $\beta^{\mathrm{old}}=(X^TWX)^{-1}X^TWX\beta^{\mathrm{old}}$ and $X^T(y-\pi)=X^TW\,W^{-1}(y-\pi)$, which gives $(X^TWX)^{-1}X^TWz$.
Weighted least squares: setting the gradient of $(z-X\beta)^TW(z-X\beta)$ to zero gives $X^TW(z-X\beta)=0$, solved by $(X^TWX)^{-1}X^TWz$. So each Newton step is that fit.
Gradient descent, from the lecture notes, skips the Hessian: $w_{t+1}=w_t-\gamma\nabla f(w_t)$ with $f=-l$. Each step is cheaper, but many more steps are needed.
Newton on the score of an intercept-only model, 2 failures in 10 welds: $\textcolor{#1f6feb}{l'(\beta_0)=2-10\pi}$. From $\beta_0=0$ the $\textcolor{#d1690a}{\text{tangent}}$ reaches zero at $-1.2$; the next tangent, dashed, reaches zero at $-1.377$, already next to the root $\log(0.2/0.8)=-1.386$.
Looks like this, but is not
Gradient ascent, $\beta\leftarrow\beta+\gamma\nabla l(\beta)$, does the same job and needs no Hessian.
It climbs, but with a fixed $\gamma$ it needs many more steps. On the 10 welds, $\gamma=0.25$ is still $0.003$ from the root after 11 steps; Newton is within $0.00003$ after 3. A gradient step is cheaper, but not per digit of accuracy.
step t
β₀
β₁
l(β)
0
$0$
$0$
$-4.159$
1
$-2.800$
$0.800$
$-2.606$
2
$-3.884$
$1.110$
$-2.485$
3
$-4.221$
$1.206$
$-2.478$
4
$-4.249$
$1.214$
$-2.478$
The log-likelihood rises at every step, and by $t=4$ the coefficients stop changing in the third decimal.
Newton steps for an intercept alone: 2 failures in 10 welds
Fit $\pi=\frac{1}{1+e^{-\beta_0}}$ to 10 welds with 2 failures by Newton's method from $\beta_0=0$. Take three steps and compare with the exact maximizer.
Find$\beta_0^{(1)},\beta_0^{(2)},\beta_0^{(3)}$ and the exact maximizer.
Given
$n=10$, $\sum_iy_i=2$
$l(\beta_0)=2\beta_0-10\log(1+e^{\beta_0})$
Solution
One coefficient, so each Newton step is a single division $l'/l''$; the exact answer is known, which lets us watch the error shrink.
Errors against $-1.386294$: $0.186$, then $0.0094$, then $0.000026$. Each error is roughly a constant times the square of the one before, the signature of Newton's method.
Three divisions for four correct decimals; gradient ascent with $\gamma=0.25$ is still off in the third decimal after 11 steps.
Once Newton is close, each step roughly doubles the number of correct digits.
One Newton step for the six students, starting from zero
Take one Newton step for the six students from $\beta^{(0)}=(0,0)$.
Find$\beta^{(1)}$.
Given
$x=1,2,3,4,5,6$; $y=0,0,1,0,1,1$
$X$ has rows $[1,x_i]$
Solution
At $\beta=0$ every $\pi_i=0.5$ and every weight is $0.25$, so $X^TWX$ is a quarter of $X^TX$, which we can write down from sums.
Same answer as the matrix step above, and also four times the least squares fit of $y-0.5$ from the least squares section: $4\cdot(-0.7,\,0.2)=(-2.8,\,0.8)$.
An IRLS step is a least squares fit you already know how to do, applied to a response that the current fit keeps redefining.
Checkpoint
§07.3 — one Newton step in one variable
A one-coefficient logistic model is being fitted by Newton's method. At the current value $\beta_0=0$ the software reports the first and second derivatives of the log-likelihood.
Find(a) What is the next iterate $\beta_0^{(1)}$?
Given
$l'(0)=-3$
$l''(0)=-2.5$
Hint 1/4
Newton moves to where the tangent of $l'$ crosses zero; the size of the move depends on both derivatives.
Each frame is one Newton step for the six students: the fitted curve $\textcolor{#1f6feb}{\pi(x)}$ steepens and settles, and the log-likelihood rises from $-4.159$ to $-2.478$. The dashed line is the $\textcolor{#d1690a}{0.5}$ cut.
At the edges
t = 0 β = (0, 0)
Every student gets probability 0.5, so the curve is flat.
t = 4 β = (−4.249, 1.214)
The maximum likelihood fit; further steps change nothing in the third decimal.
7.4Penalized logistic regression: ℓ1 and ℓ2 penalties on the log-likelihood
Subtracts a ridge or lasso penalty from the log-likelihood; keeps slopes finite even when a line separates the classes.
Ridge and the lasso added a penalty to a sum of squares that we minimized. Here the same two penalties are subtracted from a log-likelihood that we maximize.
DefinitionDefinition 7.4: ℓ1- and ℓ2-regularized logistic regression
Conditions
$\lambda\ge0$, chosen by validation as for ridge and the lasso
the intercept $\beta_0$ is not penalized: the sums start at $j=1$
inputs on comparable scales, for example standardized
Trade some log-likelihood for smaller slopes. As $\lambda\to0$ we are back at maximum likelihood, when it exists. As $\lambda\to\infty$ every slope goes to $0$ and, because the intercept is free, $\pi(x)$ becomes the training proportion of ones. The $\ell_1$ penalty can set slopes exactly to $0$; the $\ell_2$ penalty only shrinks them.
Proof
Large $\lambda$: any nonzero slope costs more and more, so the maximizer's slopes tend to $0$. With all slopes at $0$, $l=\sum_i\big[y_i\beta_0-\log(1+e^{\beta_0})\big]$, which is largest where $\frac{1}{1+e^{-\beta_0}}=\bar y$.
$\ell_2$ with Newton: the penalty adds $-2\lambda\beta_j$ to the gradient and $-2\lambda$ to the Hessian's diagonal for $j\ge1$, so the step uses $X^TWX+2\lambda D$, where $D$ is the identity with its first diagonal entry set to $0$.
$\ell_1$ has a corner at $0$. A slope stays exactly at $0$ when the log-likelihood's partial derivative there is at most $\lambda$ in size, the same test as for the lasso.
Two labelled points, $x=-1$ with $y=0$ and $x=1$ with $y=1$, which a line separates. The $\textcolor{#d1690a}{\ell_1}$ slope is $\log\frac{2-\lambda}{\lambda}$ and hits exactly $0$ at $\lambda=1$; the $\textcolor{#1f6feb}{\ell_2}$ slope solves $\frac{1}{1+e^{\beta_1}}=\lambda\beta_1$ and stays positive. With no penalty there is no best slope at all.
Looks like this, but is not
Penalize the intercept too, so that everything shrinks evenly.
Then a large $\lambda$ drives every probability to $0.5$, whatever the class balance. With 30 ones among 100 labels, a free intercept keeps the constant prediction at the sensible $0.3$.
Two points a line separates: the log-likelihood never reaches a maximum
Further reading. Fit $\operatorname{logit}\pi=\beta_0+\beta_1x$ to two points, $x=-1$ with $y=0$ and $x=1$ with $y=1$. Show that no finite $\beta_1$ maximizes the log-likelihood.
Find$l(\beta_0,\beta_1)$ and how it behaves as $\beta_1$ grows.
Given$(x,y)=(-1,0)$ and $(1,1)$
Solution
Symmetry fixes $\beta_0$ first, which leaves a function of one variable that we can study completely.
Fix the intercept by symmetry
$$\frac{\partial l}{\partial\beta_0}=\big(0-\pi(-1)\big)+\big(1-\pi(1)\big)=0\ \text{ at }\beta_0=0$$
With $\beta_0=0$, $\pi(-1)=1-\pi(1)$ for every $\beta_1$, so the residuals cancel; $l$ is concave in $\beta_0$, so this is its best value.
The first point adds $\log(1-\pi(-1))$, the second $\log\pi(1)$; both equal $-\log(1+e^{-\beta_1})$.
$$\frac{d}{d\beta_1}\,l(0,\beta_1)=\frac{2}{1+e^{\beta_1}}>0\ \text{ for every }\beta_1$$
The function keeps increasing toward $0$ and never gets there.
Answer $$\boxed{\begin{aligned}&l(0,\beta_1)=-2\log(1+e^{-\beta_1})\ \text{increases for every }\beta_1\\ &\text{no maximum}\end{aligned}}$$
Check
Values: $l(0,1)=-0.627$, $l(0,5)=-0.013$, $l(0,10)=-0.00009$. Always below $0$ and always rising, as the derivative said.
Whenever a line separates the classes, the fit can always gain by steepening; Newton's iterates then run off, and a penalty is the standard cure.
An ℓ1 penalty on the same two points: an exact zero once λ ≥ 1
Further reading. For the same two points, maximize $l(0,\beta_1)-\lambda\lvert\beta_1\rvert$ with the intercept at its symmetric value $0$. Find the penalized slope for $\lambda=0.5$ and the smallest $\lambda$ that makes it exactly $0$.
Find$\hat\beta_1$ as a function of $\lambda$, its value at $\lambda=0.5$, and the $\lambda$ where it reaches $0$.
At $\lambda=0.5$ the log-likelihood's slope at $\beta_1=\log3$ is $\frac{2}{1+3}=0.5=\lambda$, so the penalized objective is flat there, as a maximum requires.
The corner of the $\ell_1$ penalty holds a slope at exactly $0$ once $\lambda$ reaches the log-likelihood's slope there, the same mechanism as the lasso.
Checkpoint
§07.4 — a huge ℓ2 penalty with a free intercept
A churn model is fitted by $\ell_2$-regularized logistic regression with an unpenalized intercept. In the training data, 30 of 100 customers left, $y=1$. The penalty weight is made very large.
Find(a) As $\lambda\to\infty$, what does the fitted $\pi(x)$ approach for every customer?
7.5More than two classes: K − 1 log-odds against a baseline class
Gives each non-baseline class a linear score against the baseline and turns the scores into K probabilities that add to 1.
Three illnesses cannot be coded $1,2,3$ without inventing an order. Each one can, however, be compared with one fixed class through the two-class log-odds, and that needs no order.
DefinitionDefinition 7.5: Logistic regression for K classes, baseline K
Conditions
$Y\in\{1,\dots,K\}$; given $x$, the label is one draw from $K$ categories (the lecture notes' multinoulli)
$\theta=\{\beta_1,\dots,\beta_{K-1}\}$, that is $(K-1)(p+1)$ numbers
Each class except the baseline gets its own linear score against the baseline. Exponentiate the scores, count the baseline as $e^0=1$, and divide each by the total: all $K$ probabilities are positive and add to $1$. The model is fitted by maximum likelihood with Newton's method, much as for two classes.
Proof
From the $K-1$ log-odds, $\pi_j=e^{\beta_j^Tx}\,\pi_K$ for every $j<K$.
The $K$ probabilities add to $1$: $\pi_K\big(1+\sum_{l=1}^{K-1}e^{\beta_l^Tx}\big)=1$, which gives $\pi_K$, and then each $\pi_j$.
Each $\pi_j$ is a positive number over a larger positive number, so it lies in $(0,1)$, and the numerators add up to the common denominator, so the sum is exactly $1$.
An activity tracker with $K=3$: $\textcolor{#1f6feb}{\text{walk}}$ ($j=1$, $\beta_1=(-1,2)$), $\textcolor{#8250df}{\text{run}}$ ($j=2$, $\beta_2=(-6,4)$) and rest, the baseline. The most probable class changes at the $\textcolor{#d1690a}{\text{boundaries}}$ $x_1=0.5$ and $x_1=2.5$; at every reading the three curves add up to $1$.
Looks like this, but is not
Fit three separate two-class models, each class against the rest, and report their three probabilities.
Those come from three unrelated fits and need not add up to $1$. The baseline form ties the $K-1$ scores into one model whose probabilities always sum to $1$.
Resting, walking or running from one acceleration reading
An activity tracker uses $K=3$ classes, walk ($j=1$), run ($j=2$) and rest (the baseline, $j=3$), with $x=[1,x_1]^T$, $\beta_1=(-1,2)$ and $\beta_2=(-6,4)$. Find the three probabilities at $x_1=1$ and the predicted class.
Find$\pi_1,\pi_2,\pi_3$ at $x_1=1$ and the most probable class.
Given
$\beta_1=(-1,\,2)$, $\beta_2=(-6,\,4)$
$x=[1,\,1]^T$
Solution
Compute the two scores, exponentiate, and share one denominator; the baseline contributes the $1$.
Two coefficient vectors, each with an intercept and one slope.
Answer $$\boxed{\begin{aligned}&\text{rest for }x_1<0.5,\qquad \text{walk for }0.5<x_1<2.5\\ &\text{run for }x_1>2.5,\qquad 4\ \text{coefficients}\end{aligned}}$$
Check
At $x_1=2.5$ the walk and run scores are both $4$ and rest's is $0$, so $\pi_1=\pi_2=\frac{e^4}{1+2e^4}=0.495$: a tie, as a boundary requires.
With one input the boundaries are points, but not every pairwise tie is a boundary; check which class wins at each tie.
Checkpoint
§07.5 — counting coefficients in a K-class model
A handwriting app sorts strokes into $K=4$ classes using $p=3$ features plus an intercept, with class 4 as the baseline.
Find(a) How many coefficients does the model have?
Given
$K=4$ classes, baseline class $4$
$p=3$ features and an intercept
Hint 1/4
Count coefficient vectors first, then the entries of each.
Hint 2/4
There is one vector $\beta_j$ for each non-baseline class, $j=1,\dots,K-1$, each with $p+1$ entries.
Hint 3/4
Here $K-1=3$ vectors and $p+1=4$ entries each.
Hint 4/4
$3\cdot4=12$ coefficients.
Show solution
The baseline's score is fixed at 0, so it needs no vector.
Count
$$(K-1)(p+1)=3\cdot4=12$$
Three vectors, each with an intercept and three slopes.
Answer $$\boxed{12}$$
Check
With $K=2$ the same formula gives $p+1$, the two-class model's count.
Every extra class adds one more coefficient vector, not one more coefficient.
⚠ Leaving the baseline out of the denominator
the numerators run over $j<K$, so the sum is written over the same range and the $1$ is dropped
7.6Generalized linear models: a distribution, a linear predictor and a link
Names the three parts that linear, logistic and Poisson regression share: a distribution, a linear predictor and a link.
Look back at the two models we have: each has a distribution for $Y$, a linear score in $x$, and a function tying that score to the mean of $Y$.
DefinitionDefinition 7.6: Generalized linear model
Conditions
$Y\mid x$ has an exponential-family distribution: $\theta$ is the canonical parameter, $\phi$ the dispersion parameter, and $a$, $b$, $c$ are known functions
the Gaussian, Bernoulli, binomial and Poisson distributions all have this form
links: identity for linear regression, logit for logistic regression, log for Poisson regression
Pick the response's distribution from the exponential family, build the linear predictor $\eta=\beta^Tx$, and choose a link $g$ so that $g$ of the mean equals $\eta$. Linear regression is Gaussian with the identity link, logistic regression Bernoulli with the logit, Poisson regression Poisson with the log. All of them are fitted by maximum likelihood with Newton-Raphson.
Proof
The Gaussian in this form, after expanding the square: the density $$\frac{1}{\sqrt{2\pi\sigma^2}}\,e^{-\frac{(y-\mu)^2}{2\sigma^2}}$$ equals $$\exp\Big(\frac{y\mu-\mu^2/2}{\sigma^2}-\frac{y^2}{2\sigma^2}-\tfrac12\log(2\pi\sigma^2)\Big).$$
Linear regression sets the mean itself equal to $\beta^Tx$, so its link is $g(\mu)=\mu$, the identity.
The same linear predictor $\eta$ through three inverse links: the $\textcolor{#8250df}{\text{identity}}$ allows any mean, the $\textcolor{#1f6feb}{\text{logistic}}$ function keeps it between $0$ and $1$, and the $\textcolor{#1a7f37}{\text{exponential}}$ keeps it positive. The link is chosen so that the mean lands where the response's mean can live.
Looks like this, but is not
Least squares on $\log y_i$ is a log link.
That transforms the data, not the mean. It models $E[\log Y]$, which is not $\log E[Y]$, and it cannot use an observation $y_i=0$. A log link keeps the data as they are and models $\log E[Y\mid x]$.
The Bernoulli distribution in exponential-family form
Write $\Pr(Y=y)=p^y(1-p)^{1-y}$, $y\in\{0,1\}$, in the form $\exp\big(\frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi)\big)$ and read off the link.
Find$\theta$, $b(\theta)$, $a(\phi)$, $c(y,\phi)$ and the link.
Given$\Pr(Y=y)=p^y(1-p)^{1-y}$, $y\in\{0,1\}$
Solution
Take the log first: the exponential-family form says that $\log p(y)$ is linear in $y$, up to a term in $y$ alone.
Further reading check: $b'(\theta)=\frac{e^{\theta}}{1+e^{\theta}}=p$, the Bernoulli mean, as the exponential-family identity $E[Y]=b'(\theta)$ requires.
The logit link is not an arbitrary choice: it is the 's own canonical parameter.
The Poisson distribution in exponential-family form
Write $\Pr(Y=y)=\frac{\mu^y}{y!}e^{-\mu}$, $y=0,1,2,\dots$, in exponential-family form and read off the link.
Find$\theta$, $b(\theta)$, $a(\phi)$, $c(y,\phi)$ and the link.
Given$\Pr(Y=y)=\frac{\mu^y}{y!}e^{-\mu}$
Solution
The same move as for the Bernoulli: take the log and collect what multiplies $y$.
Take the log
$$\log\Pr(Y=y)=y\log\mu-\mu-\log y!$$
Three terms: $y$ times a parameter, a parameter alone, $y$ alone.
Further reading check: $b'(\theta)=e^{\theta}=\mu$, the Poisson mean.
Once $\log p(y)$ is split into $y\theta$, a function of $\theta$ and a function of $y$, the link can be read off; the same move works for the binomial.
Checkpoint
§07.6 — the link of Poisson regression
A GLM for the number of typos per page uses a Poisson response and the linear predictor $\eta=\beta^Tx$.
Find(a) Which equation ties the mean $\mu=E[Y\mid x]$ to $\eta$?
Given
$Y\mid x\sim\mathrm{Poisson}(\mu)$
$\eta=\beta^Tx$
Hint 1/4
The link must turn a positive mean into any real number, because $\eta$ can be anything.
Hint 2/4
The Poisson canonical parameter is $\theta=\log\mu$, and a GLM with this link sets $g(\mu)=\eta$.
Hint 3/4
Here $\mu=E[Y\mid x]>0$, while $\eta=\beta^Tx$ ranges over all real numbers.
Hint 4/4
$\log\mu=\eta$, that is $\mu=e^{\beta^Tx}$.
Show solution
Ask which function maps the possible means onto the possible predictors.
Match the ranges
$$\log:(0,\infty)\to(-\infty,\infty)$$
Positive means become any real number, and back through the exponential.
The log of the expected count is linear in the inputs, so one unit of $x_j$ multiplies the expected count by $e^{\beta_j}$, a rate ratio. At the fit the fitted counts reproduce the observed ones, in total and along every column of $X$. Like the logistic log-likelihood, this one is concave.
Proof
Likelihood: $\prod_i\frac{\mu_i^{y_i}}{y_i!}e^{-\mu_i}$ with $\mu_i=e^{x_i^T\beta}$. Its log is $\sum_i\big(y_ix_i^T\beta-e^{x_i^T\beta}-\log y_i!\big)$.
Gradient: $\frac{\partial}{\partial\beta}e^{x_i^T\beta}=\mu_ix_i$, so $\nabla l=\sum_i(y_i-\mu_i)x_i=X^T(y-\mu)$.
Hessian: $-\sum_i\mu_ix_ix_i^T=-X^T\mathrm{diag}(\mu)X$, negative semidefinite, so $l$ is concave, and Newton's update is $\beta+(X^TWX)^{-1}X^T(y-\mu)$ with $W=\mathrm{diag}(\mu_i)$.
The terms $\log y_i!$ do not involve $\beta$ and drop out of both derivatives.
Dropped calls in 200 hours of a call centre: the observed numbers of hours with $0,1,2,3,4$ drops, in grey, against the $\textcolor{#1a7f37}{\text{expected}}$ numbers under the fitted Poisson mean $\hat\mu=0.6$ per hour. One fitted rate reproduces the whole histogram closely.
Looks like this, but is not
Least squares on the raw counts also gives an expected count for every input.
A straight line can predict a negative count, and it gives every hour the same weight although a Poisson count's variance equals its mean, so hours with a high mean scatter more. Poisson regression builds both facts into the model.
Dropped calls: one rate for all 200 hours
A call centre records dropped calls in each of 200 hours: 109 hours with 0, 66 with 1, 21 with 2 and 4 with 3. Fit the intercept-only Poisson regression $\log E[Y]=\beta_0$, and compare the expected number of hours with each count to the observed one.
Find$\hat\beta_0$, $\hat\mu$ and the expected numbers of hours with $0$, $1$, $2$ and $3$ drops.
Given
counts $0,1,2,3$ in $109,66,21,4$ hours; $n=200$
$\log E[Y]=\beta_0$
Solution
With only an intercept the score equation is one line, so we solve it exactly instead of iterating.
The counts' variance, $0.57$, is close to their mean, $0.6$, as a Poisson count's must be, and the observed hours differ from the expected ones by at most $1.3$.
An intercept-only Poisson regression is the sample mean on the log scale; inputs then let the rate differ from hour to hour.
Weekday and weekend hours: a binary predictor in Poisson regression
Further reading. Split the same 200 hours: 150 weekday hours with 75 dropped calls and 50 weekend hours with 45. Fit $\log E[Y\mid x]=\beta_0+\beta_1x$ with $x=1$ for weekend hours, and interpret $e^{\hat\beta_1}$.
Find$\hat\beta_0$, $\hat\beta_1$ and the rate ratio.
Given
weekday, $x=0$: $150$ hours, $75$ drops
weekend, $x=1$: $50$ hours, $45$ drops
Solution
As in the logistic case, each score equation becomes a group total, so the fit reproduces the two group means.
The fitted totals are $150\cdot0.5=75$ and $50\cdot0.9=45$; together $120$ drops over $200$ hours, the $0.6$ of the one-rate fit.
$e^{\hat\beta_1}=1.8$ is a rate ratio: a weekend hour drops 18 calls for every 10 in a weekday hour, on average.
One Newton step for a Poisson regression
A server logs errors on three days with load level $x=0,1,2$: $y=1,1,2$ errors. Take one Newton step for $\log E[Y\mid x]=\beta_0+\beta_1x$ from $\beta=(0,0)$.
Find$\beta^{(1)}$.
Given
$x=0,1,2$; $y=1,1,2$
$W=\mathrm{diag}(\mu_i)$ and $\nabla l=X^T(y-\mu)$
Solution
At $\beta=0$ every $\mu_i=1$, so $W=I$ and the step needs only $X^TX$, as in the logistic step from zero.
Determinant $15-9=6$; the step is added to $\beta=0$.
Answer $$\boxed{\beta^{(1)}=(-0.167,\ 0.5)}$$
Check
The exact fit solves $X^Ty=X^T\hat\mu$: with $r=e^{\beta_1}$ the two equations reduce to $3r^2-r-5=0$, so $\hat\beta=(-0.145,\,0.384)$. The log-likelihood rises from $-3.693$ to $-3.403$ after one step; its maximum is $-3.353$.
Poisson and logistic regression share the Newton step; only the weights change, $\mu_i$ instead of $\pi_i(1-\pi_i)$.
Checkpoint
§07.7 — an expected count from the linear predictor
A Poisson regression for the number of bugs found per code review uses the review length $x$ in hundreds of lines: $\log E[Y\mid x]=0.2+0.3x$.
Find(a) What is the expected number of bugs at $x=2$?
Given
$\log E[Y\mid x]=0.2+0.3x$
$x=2$
Hint 1/4
The model gives the log of the mean; find the linear predictor, then undo the log.
Hint 2/4
$E[Y\mid x]=e^{\beta_0+\beta_1x}$.
Hint 3/4
Here $\beta_0=0.2$, $\beta_1=0.3$ and $x=2$, so $\eta=0.2+0.6=0.8$.
Hint 4/4
$E[Y\mid x=2]=e^{0.8}=2.226$.
Show solution
Linear predictor first, then the inverse link.
Linear predictor
$$\eta=0.2+0.3\cdot2=0.8$$
The log of the mean.
Mean
$$E[Y\mid x=2]=e^{0.8}=2.226$$
Undo the log.
Answer $$\boxed{2.226}$$
Check
Rate-ratio route: $E[Y\mid x=0]=e^{0.2}=1.2214$, and two steps of the rate ratio $e^{0.3}$ multiply it by $e^{0.6}=1.8221$: $1.2214\cdot1.8221=2.2255$.
In a log-link model, compute on the log scale and exponentiate once, at the end.
⚠ Forgetting to exponentiate
the linear predictor is the last thing written down
wrong$$E[Y\mid x]=\beta^Tx$$
right$$E[Y\mid x]=e^{\beta^Tx}$$
⚠ Reading the rate ratio as an added count
in linear regression a coefficient adds to the prediction
wrong$$E[Y\mid x_j+1]=E[Y\mid x_j]+e^{\beta_j}$$
right$$E[Y\mid x_j+1]=e^{\beta_j}\,E[Y\mid x_j]$$
From coefficients to a probability and a decision
A question gives fitted logistic coefficients and an input, and asks for a probability, odds, an odds ratio or a classification.
Score
Compute $\eta=\beta^Tx$, with the leading $1$ for the intercept.
Probability
$\pi=\frac{1}{1+e^{-\eta}}$; the odds are $e^{\eta}$.
Threshold on the score
For a rule $\pi\ge t$, compare $\eta$ with $\operatorname{logit}(t)=\log\frac{t}{1-t}$; $t=0.5$ means $\eta\ge0$.
Threshold on the input
With one input, solve $\beta_0+\beta_1x=\operatorname{logit}(t)$ for $x$.
Interpret
One unit of $x_j$ multiplies the odds by $e^{\beta_j}$, with the other inputs held fixed.
Where it goes wrong
Reporting the score or the odds as the probability.
Solving $\beta_0+\beta_1x\ge\operatorname{logit}(t)$ without flipping the inequality when $\beta_1<0$.
Reading $e^{\beta_j}$ as a change in probability.
One Newton (IRLS) step for logistic regression by hand
A question gives a small data set and a current $\beta$ and asks for the next iterate, or asks you to show that the update is weighted least squares.
Scores and probabilities
$\eta_i=\beta^Tx_i$ and $\pi_i=\frac{1}{1+e^{-\eta_i}}$.
Gradient
$g=X^T(y-\pi)$: the sum of the residuals, then the residuals weighted by each input.
Weights
$w_i=\pi_i(1-\pi_i)$ and $X^TWX=\sum_iw_ix_ix_i^T$.
Solve and add
$\beta^{\mathrm{new}}=\beta+(X^TWX)^{-1}g$; for two coefficients use the $2\times2$ inverse.
Check
$l$ should increase; if it falls, look for a sign error or a wrong weight, or damp the step.
Where it goes wrong
$w_i=\pi_i$ instead of $\pi_i(1-\pi_i)$.
Subtracting the step instead of adding it.
Reusing the old $\pi_i$ at the next iteration.
Recognize a GLM: canonical parameter and link
A question gives a distribution and asks whether it is in the exponential family, or which link a model uses.
Log
Write $\log p(y)$ and collect every term that multiplies $y$.
Canonical parameter
The coefficient of $y$ is $\theta/a(\phi)$; multiply it by $a(\phi)$ to get $\theta$ (Gaussian: $\frac{\mu}{\sigma^2}\cdot\sigma^2=\mu$).
The rest
Terms with the parameter alone give $-b(\theta)/a(\phi)$; terms with $y$ alone give $c(y,\phi)$.
Link
Write $\theta$ as a function of the mean $\mu$; setting it equal to $\eta=\beta^Tx$ gives the link $g$.
Where it goes wrong
Stopping while a term still mixes $y$ and the parameter nonlinearly: the form has not been reached.
Writing the inverse link as the link, for example $e^{\mu}$ for Poisson.
A yes/no response: logistic regression
Model whether a server goes down at least once on a day with load $x$: $\operatorname{logit}\pi=-2+0.5x$. Find the probability at $x=2$.
Find$\pi(2)$.
Given
$\operatorname{logit}\pi=-2+0.5x$
$x=2$
Solution
A yes/no label needs a probability, so the inverse link is the logistic function.
Odds route: $e^{-1}=0.368$ and $0.368/1.368=0.269$.
A probability can never exceed 1, whatever the load.
A count response: Poisson regression
Model how many times the server goes down on such a day: $\log E[Y\mid x]=-2+0.5x$. Find the expected count at $x=2$ and the implied probability of at least one outage.
Find$E[Y\mid x=2]$ and $\Pr(Y\ge1)$.
Given
$\log E[Y\mid x]=-2+0.5x$
$x=2$
Solution
A count needs a positive mean, so the inverse link is the exponential.
For small $\mu$, $1-e^{-\mu}\approx\mu-\mu^2/2=0.300$, close to $0.308$.
The same score gives a different number because a different question is asked of it.
The same linear score $-1$ gives a probability $0.269$ under the logit link and an expected count $0.368$ under the log link; even the Poisson model's chance of at least one outage, $0.308$, differs from the logistic answer.
How to tell them apart
Ask what one observation is: a yes/no label calls for logistic regression, a count with no fixed upper limit calls for Poisson regression.
Doubling the odds from an even chance
A coefficient $\beta_1=\log2$ doubles the odds per unit of $x$. Starting from $\pi=0.5$, find $\pi$ one unit later.
Find$\pi(x+1)$.
Given
$e^{\beta_1}=2$
$\pi(x)=0.5$
Solution
Work on the odds scale, where the coefficient acts, then convert back.
The log-likelihood went up; had it gone down, a damped step would be the fix.
Answer $$\boxed{\beta^{(1)}=(-1.2,\ 0.8)}$$
Check
Adjusted-response route: $z=4y-2=(-2,2,-2,2)$, and the least squares line through $(x_i,z_i)$ has slope $\frac{\sum(x_i-1.5)z_i}{\sum(x_i-1.5)^2}=\frac45=0.8$ and intercept $0-0.8\cdot1.5=-1.2$.
From zero, one Newton step is four times the least squares fit of $y-0.5$.
2 · you write the reasoning
Easier, and this time you write the reasons. Four emails, three of them spam; the model has an intercept only, $\pi=1/(1+e^{-\beta_0})$. Take one Newton step from $\beta_0=0$ and compare with the exact fit. For each line, write why it is allowed.
The gradient is $\sum_i(y_i-\pi)$ with $\sum_iy_i=3$ and $n=4$; the second derivative is minus the sum of the weights $\pi(1-\pi)$.
$\pi(0)=0.5:\quad l'(0)=1,\qquad l''(0)=-1$
reasoning
A zero score gives $\pi=0.5$ and a weight of $0.25$ for each of the four emails.
$\beta_0^{(1)}=0-\frac{1}{-1}=1$
reasoning
Newton's update $\beta_0-l'/l''$.
$\hat\beta_0=\log\frac{0.75}{0.25}=\log3=1.099$
reasoning
The exact fit sets $\pi=\bar y=0.75$; one step got within $0.1$ of it.
3 · find the buried error
Harder, with two errors buried in the solution. Data $x=(-1, \allowbreak -1, \allowbreak 1, \allowbreak 1, \allowbreak 1)$ and $y=(0,1,0,1,1)$; model $\operatorname{logit}\pi=\beta_0+\beta_1x$. A student takes one Newton step from $\beta=(0,\,\log3)$. Which two steps are wrong?
Step 1. The scores $\mp\log3$ give $\pi=(0.25,\, \allowbreak 0.25,\, \allowbreak 0.75,\, \allowbreak 0.75,\, \allowbreak 0.75)$.
Step 2. The residuals $(-0.25,\, \allowbreak 0.75,\, \allowbreak -0.75,\, \allowbreak 0.25,\, \allowbreak 0.25)$ give $g=X^T(y-\pi)=(0.25,\,-0.75)$.
Step 3. With the weights $w_i=\pi_i$: $X^TWX=\begin{pmatrix}2.75&1.75\\1.75&2.75\end{pmatrix}$.
Step 4.$\beta^{\mathrm{new}}=\beta-(X^TWX)^{-1}g$, so $\beta^{\mathrm{new}}=(0,\,1.099)-(0.444,\,-0.556)=(-0.444,\,1.654)$.
the two buried errors (2)
⚠ step 3
The weights must be $\pi_i(1-\pi_i)$, which is $0.1875$ for every point here, not $\pi_i$.
The derivative of the logistic function, $\pi(1-\pi)$, is remembered as $\pi$ itself.
Evaluate $\sum_i[y_i\eta_i-\log(1+e^{\eta_i})]$ at both points.
Answer $$\boxed{\beta^{(1)}=(-2,\ 0.8)}$$
Check
Adjusted response: $z=4y-2=(-2, \allowbreak -2, \allowbreak 2, \allowbreak -2, \allowbreak 2)$; its least squares line on $x$ has slope $\frac{\sum(x_i-2)z_i}{\sum(x_i-2)^2}=\frac{8}{10}=0.8$ and intercept $\bar z-0.8\cdot2=-0.4-1.6=-2$.
When every weight is equal, IRLS is plain least squares on z; that shortcut works only on the first step from zero.
Full exam-style question
Welds by shift: derive the Newton update, take one step, check it against the exact fitexam format
Exam-style. Ten welds: on the day shift, $x=0$, 1 of 4 failed; on the night shift, $x=1$, 3 of 6 failed. (a) Derive $\nabla l$ and the Hessian for $\operatorname{logit}\pi=\beta_0+\beta_1x$, and show that $l$ is concave. (b) Take one Newton step from $\beta=(0,0)$. (c) Find the exact maximum likelihood fit and compare.
Find$\nabla l$, $\nabla^2l$, $\beta^{(1)}$ and $\hat\beta^{\mathrm{MLE}}$.
One step from zero is already within $0.1$ of the exact fit in both coefficients, and at $\hat\beta$ the fitted failures, $4\cdot\frac14=1$ and $6\cdot\frac12=3$, are the observed counts.
Whenever the only predictor is binary, the exact fit is two logits of group rates, so any Newton computation can be checked in one line.
Practice
A · concept 4 questions
1§07.1 — does a coefficient add to the probability?
A logistic model for whether a student finishes an online course uses weekly logins $x$: $\operatorname{logit}\pi=-1+0.7x$. A classmate reads $\beta_1=0.7$ as: each extra weekly login raises the chance of finishing by $0.7$.
Find(a) True or false: one more weekly login raises $\pi$ by $0.7$.
Given$\operatorname{logit}\pi(x)=-1+0.7x$
Hint 1/4
Decide which scale the coefficient lives on: probability, odds or log-odds.
Hint 2/4
$\operatorname{logit}\pi(x+1)-\operatorname{logit}\pi(x)=\beta_1$, so the odds are multiplied by $e^{\beta_1}$.
Hint 3/4
Here $\beta_1=0.7$, in $\operatorname{logit}\pi=-1+0.7x$. A concrete check: $\pi(1)=1/(1+e^{0.3})=0.426$ and $\pi(2)=1/(1+e^{-0.4})=0.599$.
Hint 4/4
$\pi$ rose by $0.173$, not $0.7$: the claim is false.
Show solution
One concrete pair of inputs is enough to refute a claim about every input.
The odds, not the probability, change by a fixed factor; the last digit is rounding.
Answer $$\boxed{\text{False}}$$
Check
Even the best case fails: from $\pi=0.5$ a rise of $0.7$ would give $1.2$, impossible for a probability.
Coefficients add on the log-odds scale and multiply on the odds scale; on the probability scale their effect depends on the starting point.
2§07.1 — coding three unordered classes as numbers
A maintenance team codes three fault types as bearing $=1$, winding $=2$, rotor $=3$, fits least squares on sensor features, and predicts the fault whose code is nearest the fitted value. A colleague recodes the same faults as $2,3,1$ and refits.
Find(a) What happens to the predictions?
Given
codes $(1,2,3)$ against $(2,3,1)$ for the same three faults
least squares fit, nearest-code prediction
Hint 1/4
Ask whether least squares sees the fault names or only the numbers standing for them.
Hint 2/4
The fit uses $y$ as numbers on a line: it assumes the classes are ordered and equally spaced.
Hint 3/4
Under $(1,2,3)$ winding sits between bearing and rotor; under $(2,3,1)$ bearing sits between rotor and winding.
Hint 4/4
Different orders give different fits, so the predicted faults can change.
Show solution
Looking at the order each coding implies settles the question without any data.
The fitted coefficients are linear in $y$, so a different $y$ gives different coefficients.
Answer $$\boxed{\text{the predictions can change}}$$
Check
With two classes the problem disappears: swapping $0$ and $1$ maps $\hat y$ to $1-\hat y$ and the $0.5$ cut to itself, so every prediction stays.
With three or more unordered classes, use a model built on probabilities, such as the multinomial model of this section.
3§07.2 — can Newton stop at the wrong peak?
A classmate worries that the logistic log-likelihood might have two separate peaks, so Newton's method started at $\beta=0$ could stop at the lower one.
Find(a) True or false: the logistic log-likelihood can have two separate local maxima.
Given$\nabla^2l(\beta)=-X^TWX$ with $W=\mathrm{diag}\big(\pi_i(1-\pi_i)\big)$
Hint 1/4
Two separate peaks need a dip between them, which a cannot have.
Hint 2/4
$l$ is concave if $v^T\nabla^2l(\beta)\,v\le0$ for every $\beta$ and every $v$.
Hint 3/4
With $\nabla^2l=-X^TWX$: $v^T(-X^TWX)v=-\sum_i\pi_i(1-\pi_i)(x_i^Tv)^2$.
Hint 4/4
Every term is at most $0$, so $l$ is concave and has no second peak. False.
Show solution
Concavity is a statement about the Hessian, which we already have in closed form.
Quadratic form
$$v^TX^TWXv=\sum_iw_i(x_i^Tv)^2$$
$W$ is diagonal, so the quadratic form is a weighted sum of squares.
Test the others at two inputs: $\pi_1/\pi_3=e^{\beta_1^Tx}$ grows exponentially, not linearly, and $\pi_1$ stays below $1$ however large $\beta_1^Tx$ becomes.
In the K-class model every pairwise log-odds is linear, so every pairwise boundary is a line or a plane.
B · computation 8 questions
1§07.1 — loan default: two probabilities and a risk-averse cut
A bank's fitted model for missing a card payment uses the balance $x$ in TL: $\operatorname{logit}\pi=-8+0.004x$. The bank calls a customer when $\pi\ge0.1$.
Find
(a) Find $\pi$ at balances of $1500$ and $2500$ TL.
(b) From which balance does the bank call?
Given
$\operatorname{logit}\pi(x)=-8+0.004x$
call if $\pi\ge0.1$
Hint 1/4
Work with scores: probabilities come from scores, and a probability threshold is a score threshold.
Hint 2/4
$\pi=\frac{1}{1+e^{-\eta}}$ with $\eta=-8+0.004x$, and $\pi\ge t\iff\eta\ge\log\frac{t}{1-t}$.
Hint 3/4
Scores: $\eta(1500)=-2$ and $\eta(2500)=2$. Threshold: $\log\frac{0.1}{0.9}=-2.197$, with $\eta=-8+0.004x$.
Hint 4/4
$\pi(1500)=0.119$, $\pi(2500)=0.881$; the bank calls from $x\approx1451$ TL.
Show solution
Every question here is easiest on the score scale, where the model is linear.
From $x=2000$, where $\pi=0.5$ and the odds are $1$, to $x=2173.3$: the score is $0.693$ and the odds $e^{0.693}=2.00$.
The odds-doubling distance, $\log2/\beta_1$, is the same everywhere on the curve; the change in probability is not.
3§07.2 — side effects under two drugs, fitted in closed form
Further reading. Of 200 patients given drug A, 30 reported a side effect; of 100 given drug B, 25 did. Model $\operatorname{logit}\pi=\beta_0+\beta_1x$ with $x=1$ for drug B.
Find
(a) Find $\hat\beta_0$ and $\hat\beta_1$.
(b) Give the odds ratio of B against A.
Given
drug A, $x=0$: $30$ of $200$
drug B, $x=1$: $25$ of $100$
Hint 1/4
With one binary predictor the maximum likelihood fit reproduces each group's rate; find the two rates first.
Hint 2/4
$\hat\beta_0=\operatorname{logit}(\hat\pi_A)$ and $\hat\beta_1=\operatorname{logit}(\hat\pi_B)-\operatorname{logit}(\hat\pi_A)$.
Hint 3/4
Rates: $\hat\pi_A=30/200=0.15$ and $\hat\pi_B=25/100=0.25$.
Hint 4/4
$\hat\beta_0=-1.735$, $\hat\beta_1=0.636$, odds ratio $1.889$.
Show solution
The score equations for a binary predictor say that each group's fitted failures equal its observed ones.
At $\beta_0^{(2)}$ the fitted spam rate is $1/(1+e^{1.0963})=0.2504$, within $0.0005$ of the observed $0.25$.
The intercept-only fit is always the logit of the sample proportion; Newton gets there in a few steps from zero.
5§07.3 — a Newton step with a diagonal matrix
Five parts are tested at centred temperatures $x=-2,-1,0,1,2$, and $y=1$ marks a failure: $y=(0,1,0,0,1)$. Fit $\operatorname{logit}\pi=\beta_0+\beta_1x$.
The exact fit is $(-0.444,\,0.439)$, so the first step is already within $0.05$ in both coefficients.
Centring an input decouples intercept and slope in the first step, which makes the arithmetic one division per coefficient.
6§07.5 — three topic probabilities with two features
A news classifier uses the classes sports ($j=1$), economy ($j=2$) and other (the baseline, $j=3$), with $x=[1,x_1,x_2]^T$, $\beta_1=(0.5,\,1,\,-1)$ and $\beta_2=(-0.5,\,0,\,2)$.
The three add to $0.999$, one up to rounding, and $\pi_1/\pi_2=1.649=e^{1-0.5}$, as the pairwise log-odds requires.
With more features only the scores get longer; the rest of the computation is the same.
7§07.7 — expected calls and a weekend rate ratio
A Poisson regression for calls to a help line per hour is $\log E[Y\mid x]=0.2+0.3x_1-0.5x_2$, where $x_1$ is the number of open tickets in tens and $x_2=1$ on weekends.
Find
(a) Find the expected number of calls in that hour.
(b) Give the weekend rate ratio and say what it means.
(c) Find the probability of no calls in that hour.
Given
$\log E[Y\mid x]=0.2+0.3x_1-0.5x_2$
an hour with $x_1=2$ on a weekend
Hint 1/4
Compute the linear predictor, then undo the log; a rate ratio is a coefficient on the multiplicative scale.
Hint 2/4
$\mu=e^{\eta}$, the rate ratio is $e^{\beta_2}$, and $\Pr(Y=0)=e^{-\mu}$.
Hint 3/4
$\eta=0.2+0.3\cdot2-0.5\cdot1=0.3$, and $\beta_2=-0.5$.
Hint 4/4
$\mu=1.350$, rate ratio $0.607$, $\Pr(Y=0)=0.259$.
Show solution
Everything follows from the linear predictor and the Poisson pmf at zero.
Further reading check: $b'(\theta)=\frac{me^{\theta}}{1+e^{\theta}}=mp$, the binomial mean; with $m=1$ everything reduces to the Bernoulli form.
Counting successes out of a known number of trials gives logistic regression again, with the count in place of a single label.
C · exam level 5 questions
1§07.3 — the adjusted response and the weights of one IRLS step
During an IRLS fit, three points have current scores $\eta=\log\frac14,\ 0,\ \log4$, so their probabilities are $0.2$, $0.5$ and $0.8$. Their labels are $y=0,1,1$.
Find
(a) Find the weights $w_i$ and the adjusted responses $z_i$.
(b) Which point has the largest weight in the next weighted least squares fit, and why?
Given
$\eta=(-1.386,\,0,\,1.386)$ and $\pi=(0.2,\,0.5,\,0.8)$
$y=(0,\,1,\,1)$
Hint 1/4
Each point needs a weight and a target, and both come from its current probability.
Hint 2/4
$w_i=\pi_i(1-\pi_i)$ and $z_i=\eta_i+\frac{y_i-\pi_i}{w_i}$.
Hint 3/4
Point 1: $\pi=0.2$, $y=0$. Point 2: $\pi=0.5$, $y=1$. Point 3: $\pi=0.8$, $y=1$. Scores $\eta=(-1.386,\,0,\,1.386)$.
Hint 4/4
$w=(0.16,\,0.25,\,0.16)$ and $z=(-2.636,\,2,\,2.636)$; point 2 weighs the most.
Show solution
Both formulas use only each point's own numbers, so we go point by point.
Undo the adjustment: $w_i(z_i-\eta_i)$ should return the residuals $y_i-\pi_i$. Indeed $0.16\cdot(-1.25)=-0.2$, $0.25\cdot2=0.5$, $0.16\cdot1.25=0.2$.
Points the current fit already calls confidently get less weight; points near 0.5 steer the next step.
2§07.7 — Poisson score equations and a closed-form fit
Further reading in part (b). A web server logs errors per hour: 40 daytime hours with 100 errors in total and 20 night hours with 20 errors. Model $\log E[Y\mid x]=\beta_0+\beta_1x$ with $x=1$ for night.
Find
(a) Show that the maximum likelihood fit satisfies $X^Ty=X^T\hat\mu$, and say what the first equation means.
(b) Solve for $\hat\beta_0$ and $\hat\beta_1$, and interpret $e^{\hat\beta_1}$.
Day hours have $x_i=(1,0)$ and $\mu=e^{\beta_0}$; night hours have $x_i=(1,1)$ and $\mu=e^{\beta_0+\beta_1}$. Totals: $100$ errors in $40$ hours and $20$ in $20$ hours.
Hint 4/4
$\hat\beta_0=\log2.5=0.916$ and $\hat\beta_1=\log0.4=-0.916$: nights have $0.4$ times the daytime rate.
Show solution
The gradient is general; the binary predictor then turns each equation into a group total.
Fitted totals: $40\cdot2.5=100$ and $20\cdot1=20$, matching both groups exactly.
In any Poisson regression with an intercept, the fitted counts add up to the observed total.
3§07.1 — reading a negative coefficient in a multiple model
A hospital's model for readmission within 30 days uses age and $C=1$ for the city hospital: $\operatorname{logit}\pi=-5+0.05\,\text{age}-0.4C$. City patients are 70 years old on average, rural ones 50.
Answer $$\boxed{\text{better at equal age, worse for the typical patient}}$$
Check
Split the typical gap: $-1.9-(-2.5)=0.6=0.05\cdot20-0.4$; the 20-year age gap adds $1.0$ and the city takes $0.4$ away.
Before reading a coefficient's sign as good or bad, ask what is being held fixed.
4§07.4 — an ℓ1-penalized slope for two separated points
Further reading. Two points: $x=-2$ with $y=0$ and $x=2$ with $y=1$. By symmetry the intercept is $0$, and then $l(0,\beta_1)=-2\log(1+e^{-2\beta_1})$.
Find
(a) Show that without a penalty no finite slope maximizes $l$.
(b) Find the penalized slope for $\lambda=1$.
(c) Find the smallest $\lambda$ that sets the slope exactly to $0$.
Check each statement against the two update formulas in the given data.
Hint 2/4
Newton uses $X^TWX$ with $W=\mathrm{diag}\big(\pi_i(1-\pi_i)\big)$ at the current $\beta$; gradient ascent uses only $X^T(y-\pi)$.
Hint 3/4
The given updates again: Newton $\beta+(X^TWX)^{-1}X^T(y-\pi)$, with $W$ recomputed from the new $\pi$ at every step, and gradient ascent $\beta+\gamma X^T(y-\pi)$.
Hint 4/4
The statement about weighted least squares with changing weights is the right one.
Show solution
Each wrong statement contradicts one formula, so we test them one at a time.
Newton
$$\begin{aligned}&\beta^{\mathrm{new}}=(X^TWX)^{-1}X^TWz\\ &W=\mathrm{diag}\big(\pi_i(1-\pi_i)\big)\ \text{at the current }\beta\end{aligned}$$
A weighted least squares solution with weights that move with $\beta$.
The others
$$\begin{aligned}&\text{gradient ascent: no }H\\ &\text{no closed form in general}\\ &\text{step added, since }H=-X^TWX\end{aligned}$$
Each of the other three statements breaks one of these facts.
Answer $$\boxed{\text{weighted least squares with changing weights}}$$
Check
The six students' iterations used weights $0.25$ at $t=0$ and weights from $0.105$ to $0.240$ at $t=1$: the weights did change.
Newton needs more work per step and far fewer steps; gradient ascent is the reverse.
D · interleaved 5 questions
1§07.2 — a click rate three ways
An online shop showed an ad 300 times and got 45 clicks. Treat the views as independent Bernoulli trials with click probability $p$.
Find
(a) Find the maximum likelihood estimate of $p$.
(b) Fit an intercept-only logistic regression and check that it gives the same $p$.
(c) Find the MAP estimate under the Beta prior.
Given
$45$ clicks in $300$ views
prior for part (c): $p\sim\mathrm{Beta}(2,18)$, whose posterior mode is $\frac{N_1+a-1}{n+a+b-2}$
Hint 1/4
Parts (a) and (b) are the same fit on two scales; part (c) adds the prior's pseudo-counts.
Hint 2/4
$\hat p=N_1/n$; the intercept-only logistic fit solves $\sum_i(y_i-\pi)=0$; the Beta mode formula is given.
Hint 3/4
$N_1=45$, $n=300$, $a=2$, $b=18$.
Hint 4/4
(a) $0.15$. (b) $\hat\beta_0=\log\frac{0.15}{0.85}=-1.735$, and $1/(1+e^{1.735})=0.15$. (c) $46/318=0.145$.
Show solution
Maximum likelihood and the intercept-only logistic fit solve the same equation; MAP adds pseudo-counts.
Maximum likelihood
$$\hat p=\frac{45}{300}=0.15$$
The Bernoulli maximum likelihood estimate is the proportion.
The MAP estimate lies between the prior's mode $\frac{1}{18}=0.056$ and the data's $0.15$, pulled only slightly because 300 views outweigh 18 pseudo-observations.
Logistic regression with an intercept only is the Bernoulli estimate on the log-odds scale.
2§07.4 — the ℓ2 penalty as a Gaussian prior
In the Bayesian section, ridge regression was a MAP estimate under a Gaussian prior. Do the same for $\ell_2$-regularized logistic regression: put independent priors $\beta_j\sim\mathcal N(0,\tau^2)$ on the slopes and a flat prior on $\beta_0$.
Find
(a) Show that the MAP estimate maximizes $l(\beta)-\lambda\sum_{j\ge1}\beta_j^2$, and find $\lambda$ in terms of $\tau^2$.
(b) Which prior variance corresponds to $\lambda=0.5$? Why is there no $\sigma^2$ in the answer?
Compare with ridge: there the log-likelihood is $-\frac{\mathrm{RSS}}{2\sigma^2}$, and multiplying through by $2\sigma^2$ produced $\lambda=\sigma^2/\tau^2$. Here nothing is rescaled, so the factor $\frac12$ stays.
A penalty is a prior in disguise; the translation constant depends on how the log-likelihood is scaled.
3§07.1 — picking a threshold on a validation set
A fitted logistic model scores eight validation emails. The team compares three thresholds by the number of misclassified emails.
Find
(a) Count the validation errors for each threshold.
(b) Which threshold would you report, and why is it chosen on validation data rather than on the training set?
Any threshold above $0.3$ and at most $0.4$ gives the same single error, so $0.35$ is not a lucky pick.
A threshold is a tuning parameter like any other, so it is chosen on data the model did not see.
4§07.3 — the first IRLS step from least squares on the labels
In the least squares section the six students ($x=1,\dots,6$; $y=0,0,1,0,1,1$) gave the line $\hat y=-0.2+0.2x$. Use it to find the first Newton step of logistic regression from $\beta=0$ without any matrix.
Find
(a) Explain why the first IRLS step is four times the least squares fit of $y-0.5$.
(b) Give $\beta^{(1)}$.
Given
least squares on the labels: $\hat y=-0.2+0.2x$
at $\beta=0$: $\pi_i=0.5$, $w_i=0.25$, $z_i=4(y_i-0.5)$
Hint 1/4
At $\beta=0$ the weights are all equal, so the weighted least squares step is an ordinary one.
Hint 2/4
$\beta^{(1)}=(X^TWX)^{-1}X^TWz$ with $W=0.25\,I$ and $z=4(y-0.5)$; least squares is linear in the response.
Hint 3/4
Least squares of $y$ gives $(-0.2,\,0.2)$; for $y-0.5$, subtract $0.5$ from the intercept.
Hint 4/4
$\beta^{(1)}=4\cdot(-0.7,\,0.2)=(-2.8,\,0.8)$.
Show solution
Linearity of least squares in the response does all the work.
Newton weights $W=\mathrm{diag}(\mu_i)$; rate ratio $e^{\beta_j}$
Check yourself
Close the page and write down from memory:
the logistic model, its odds and its log-odds;
the log-likelihood and its gradient;
one Newton step for logistic regression, with its weights;
the K-class probabilities with a baseline class;
the three parts of a GLM and the three links;
Poisson regression's log-likelihood and score equations.
Then reopen the page and compare; whatever is missing is your reread list.
Turn coefficients into a probability, odds and a decision at a given threshold, and explain an odds ratio?
c-logistic
Derive $l(\beta)$ and $X^T(y-\pi)$, and fit a binary predictor by hand?
c-logistic-mle
Take a two-coefficient Newton step from $\beta=0$ and write it as weighted least squares?
c-newton-irls
Say what a penalized fit does as $\lambda\to0$ and $\lambda\to\infty$, with and without ?
c-penalized
Compute three class probabilities and find where the predicted class changes?
c-multinomial
Put the Bernoulli, binomial or Poisson pmf in exponential-family form and name its link?
c-glm
Fit an intercept-only or binary-predictor Poisson regression and interpret $e^{\beta_j}$?
c-poisson
Glossary (28 terms)
logistic functionlojistik fonksiyon
The map $t\mapsto\frac{e^t}{1+e^t}=\frac{1}{1+e^{-t}}$, which sends any real number into $(0,1)$ and turns a linear score into a probability.
logistic regressionlojistik regresyon
A model for a 0/1 label in which the log-odds of class 1 is linear in the inputs: $\log\frac{\pi}{1-\pi}=\beta^Tx$.
log-odds
The logarithm of the odds, $\log\frac{\pi}{1-\pi}$, also called the logit; it ranges over all real numbers and is $0$ at $\pi=0.5$.
odds
The probability of an event divided by the probability of its complement, $\pi/(1-\pi)$; odds of $4$ mean a probability of $0.8$.
odds ratioodds oranı
The ratio of the odds in two situations; in logistic regression a unit increase in $x_j$ multiplies the odds by $e^{\beta_j}$.
Bernoulli distributionBernoulli dağılımı
The distribution of a single 0/1 outcome with $\Pr(Y=1)=p$: $\Pr(Y=y)=p^y(1-p)^{1-y}$.
decision thresholdkarar eşiği
The probability $t$ above which a classifier predicts class 1; $t=0.5$ is risk-neutral, a smaller $t$ flags more cases.
confoundingkarıştırıcı etki
A situation in which an input that differs between groups also affects the response, so that a group comparison and a coefficient at fixed inputs point in different directions.
score equationsskor denklemleri
The equations obtained by setting the gradient of the log-likelihood to zero; for logistic regression $X^T(y-\pi)=0$.
concave functioniçbükey fonksiyon
A function whose Hessian is negative semidefinite everywhere; any point where its gradient vanishes is a global maximum.
Newton-RaphsonNewton Raphson yöntemi
An iterative root finder that replaces a function by its tangent and jumps to the tangent's root: $w_{t+1}=w_t-H^{-1}\nabla f(w_t)$ when applied to a gradient.
HessianHesse matrisi
The matrix of second partial derivatives of a function; for the logistic log-likelihood it is $-X^TWX$.
gradient descentgradyan inişi
An iterative method that moves against the gradient with a step size $\gamma$: $w_{t+1}=w_t-\gamma\nabla f(w_t)$; applied to $-l$ it climbs the log-likelihood.
step sizeadım büyüklüğü
The factor $\gamma$ that scales each update in gradient descent or damps a Newton step.
iteratively reweighted least squaresyinelemeli yeniden ağırlıklandırılmış en küçük kareler
Newton's method for logistic regression written as a sequence of weighted least squares fits whose weights change after every step.
weighted least squaresağırlıklı en küçük kareler
Least squares in which each squared residual carries a weight: minimize $(z-X\beta)^TW(z-X\beta)$, solved by $(X^TWX)^{-1}X^TWz$.
adjusted response
The target $z=X\beta+W^{-1}(y-\pi)$ of the weighted least squares fit inside one IRLS step.
separable dataayrılabilir veri
Data in which a line or plane puts every class-1 point on one side and every class-0 point on the other; the logistic maximum likelihood estimate then does not exist.
çok terimli lojistik regresyon
Logistic regression for $K$ unordered classes: $K-1$ linear log-odds against a baseline class, turned into $K$ probabilities that add to 1.
baseline classreferans sınıf
The class whose score is fixed at $0$ in the multinomial model; every other class's log-odds is measured against it.
generalized linear modelgenelleştirilmiş doğrusal model
A model with an exponential-family response, a linear predictor $\eta=\beta^Tx$ and a link $g$ with $g(E[Y\mid x])=\eta$.
exponential familyüstel aile
Distributions of the form $\exp\big(\frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi)\big)$, among them the Gaussian, Bernoulli, binomial and Poisson.
canonical parameterkanonik parametre
The parameter $\theta$ that multiplies $y$ in an exponential-family density; $\log\frac{p}{1-p}$ for the Bernoulli, $\log\mu$ for the Poisson.
dispersion parameteryayılım parametresi
The parameter $\phi$ of an exponential family that scales the spread, such as $\sigma^2$ for the Gaussian; for the Bernoulli and Poisson, $a(\phi)=1$.
linear predictor
The linear score $\eta=\beta^Tx$ in a generalized linear model.
link functionbağlantı fonksiyonu
The function $g$ with $g(\mu)=\eta$ that ties the mean response to the linear predictor: identity, logit or log for the three models of this section.
Poisson regressionPoisson regresyonu
A generalized linear model for counts with a Poisson response and a log link: $\log E[Y\mid x]=\beta^Tx$.
rate ratio
The factor $e^{\beta_j}$ by which a unit increase in $x_j$ multiplies the expected count in Poisson regression.
What comes next
§08 · Perceptron, neural networks and backpropagation
Here one linear score went through one fixed function. Next, layers of such scores feed into each other, so the score is no longer linear in the parameters, and fitting again means following a gradient.
Sources
textbookT. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning, Springer (course textbook) The weekly line names no chapter of the book, so no section numbers are cited here.
course materialEEE 485 lecture slides and lecture notes, chapter 7: Generalized linear models Scope, order of topics and notation (π(X), odds and logit, D, l(β), X, W, z, the adjusted response, η, μ, g, θ, φ, a, b, c, the baseline class K, the step size γ) follow these materials; all explanations, data, figures and exercises here are original.
course materialEEE 485 syllabus page on STARS, Fall 2026-27, printed 21 September 2026 Assessment weights, the weekly topic list and the recommended books.
textbookG. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning, Springer, 2013 Recommended in the syllabus; the lecture's credit card default example and its figures come from it.
standard resultNewton's method, the exponential family and the Poisson distribution Standard results used in the derivations. Every number and figure on this page was computed for it.