The same model, read as probability

Start from the model that least squares already assumes implicitly. With features φ(x) (the raw inputs, or any fixed basis expansion of them), each target is a linear function plus Gaussian noise:

Maximising this likelihood is ordinary least squares. Now add a prior on the weights: each one is zero-mean Gaussian with variance τ². It is convenient to work with precisions (inverse variances): α = 1/τ² for the prior and β = 1/σ² for the noise. Bayes' rule gives the posterior up to a constant, and its negative log is

Multiply by 2/β and this is the ridge objective ‖y − Φw‖² + λ‖w‖² with

So the ridge penalty is not an arbitrary knob: it is the ratio of the noise variance to the prior variance of the weights. Large λ means noisy data or a strong belief that weights are small. Swapping the Gaussian prior for a Laplace prior, p(w_j) ∝ exp(−|w_j|/b), turns the penalty into ‖w‖₁ and gives the lasso (with λ = σ²/b for the ½‖y − Φw‖² + λ‖w‖₁ convention).

Ridge and lasso are already Bayesian in their assumptions; they just stop at the peak. Everything else in this note comes from not stopping there.

Prior × likelihood, one weight at a time

The cleanest picture uses a single weight: a line through the origin, y = w·x + ε, with true slope 0.5. The posterior over w is the prior multiplied by the likelihood and renormalised. The ridge or lasso estimate is the location of the posterior's peak; the shaded region is the 95% credible interval, which no point estimate gives you.

Things to try:

• Set n = 0: the posterior is the prior. Add points and the likelihood narrows and drags the posterior toward the data; with n = 50 the prior barely matters.
• Shrink τ: the prior dominates and the estimate is pulled to 0 (that is the "shrinkage" in ridge). Raise σ: each point is less informative, so the same thing happens. Only the ratio σ²/τ² matters for the peak.
• Switch to the Laplace prior and shrink τ (or use few points): the peak snaps to exactly 0, which is how lasso produces sparse solutions. The posterior mean, however, is not zero: the mass on either side of the kink still pulls it away. A fully Bayesian lasso averages instead of picking the peak, and so does not give exact zeros.

The posterior in closed form

With a Gaussian prior and Gaussian noise, the posterior is again Gaussian (the prior is conjugate), so it is fully described by a mean vector and a covariance matrix. With design matrix Φ (one row φ(x_n)^T per data point):

The second form of m_N is the ridge closed form, term for term. Three readings of the covariance are worth keeping:

• Precisions add. The prior contributes αI. Each data point contributes βφ_nφ_n^T, a rank-one increase in certainty along the direction φ_n and no information in the directions orthogonal to it.
• S_N does not depend on y. How uncertain the weights are depends only on where you measured (and on σ), not on the values you observed. This is also why Bayesian linear models are used for experimental design.
• Learning is sequential. Today's posterior is tomorrow's prior: updating one point at a time, with N(m_N, S_N) as the prior for point N+1, gives exactly the same result as the batch formula.

With intercept and slope, y = w₀ + w₁x, the weight space is a plane and the posterior is an ellipse in it. The demo below draws points from the true line w = (−0.3, 0.5) one at a time. The left panel is the posterior over (w₀, w₁); the right panel shows lines drawn from it. The dashed line in weight space is the set of weights that fit the newest point exactly: each observation constrains one direction and leaves the other free, so the ellipse collapses perpendicular to that line.

With zero points the samples are the prior: random lines of every slope. After one point every sample passes near it but still pivots freely, which is the long thin ellipse. After a handful the samples agree and the ellipse closes around the true weights. The posterior correlation between w₀ and w₁ also shows a familiar effect: if all x values sit on one side of zero, raising the intercept and lowering the slope fit the data almost equally well.

The predictive distribution: where the error bars come from

For a new input x*, the Bayesian prediction averages the model's prediction over every plausible weight vector, weighted by the posterior. For this model the integral is again Gaussian:

The mean is the ridge prediction. The variance has two parts:

• σ², the noise (aleatoric) term. Irreducible; it is the same everywhere and no amount of data removes it.
• φ(x*)^TS_Nφ(x*), the model (epistemic) term. The uncertainty about w, projected onto the features at x*. It shrinks as data arrives near x* and stays large where there is none. It never increases when a point is added.

The demo fits a nonlinear function with a fixed basis: a bias plus nine Gaussian bumps spread over [0, 1]. The model is still linear in w, so every formula above applies unchanged. Tap or click the chart to add points; the data initially has a gap in the middle.

Move the probe into the gap: the model term dominates there and the band widens, while near clusters of points it is dominated by σ. Ridge regression returns only the solid line, with no signal that the middle of the curve is a guess. Turn on function samples to see the same thing as a spread of plausible curves, which agree where there is data and fan out where there is not.

Choosing λ without cross-validation: the evidence

With ridge, λ is chosen by cross-validation. The Bayesian setup offers a different route. Integrating the weights out gives the probability of the observed targets as a function of the hyperparameters alone, called the marginal likelihood or evidence:

For this model it has a closed form (M is the number of weights, N the number of points):

E(m_N) is the regularised fit, which favours flexible models. The log-determinant and ln α terms penalise how much the posterior had to shrink relative to the prior, which favours simple ones. The balance is a built-in Occam's razor: a model that could have explained many datasets spreads its probability thinly and is penalised for it, even if it fits this one well.

Maximising the evidence over α and β (called empirical Bayes, type-II maximum likelihood or the evidence approximation) has a simple fixed-point solution:

γ lies between 0 and M and counts the effective number of parameters: weights that the data determines well, as opposed to ones held near zero by the prior. The noise update is the familiar residual variance with degrees of freedom N − γ. This uses only the training data; no held-out fold is needed.

The left chart below is the log evidence for the data in the previous demo as α varies (σ fixed at its current value). The right chart compares polynomial models of increasing degree on the same data, each with its own α and σ set by maximising its evidence. The dashed line is the training error (RMSE), which only falls as the degree grows; the evidence rises while extra flexibility explains real structure, then levels off or falls once the extra terms only buy the ability to fit noise.

Add or remove points in the previous demo and both charts update. With very few points the evidence prefers low-degree polynomials and strong regularisation; as data accumulates it supports more flexibility. The evidence is a statement about the model family as well as the data: if the model is badly misspecified, maximising it can still produce overconfident error bars, so it complements rather than replaces checking predictions on held-out data.

Summary and where it leads

Extensions that follow directly from this setup:

• Unknown noise. Treat σ² as random too, with an inverse-gamma prior (the Normal-inverse-gamma conjugate prior). The predictive distribution becomes a Student-t, with heavier tails when there is little data.
• Non-Gaussian priors. A Laplace or horseshoe prior has no closed-form posterior; it needs MCMC or variational approximations. The Bayesian lasso (Park and Casella, 2008) is the Laplace case.
• One α per weight. Giving each weight its own prior precision and maximising the evidence drives many α_j to infinity, pruning those weights. This is automatic relevance determination, the basis of the relevance vector machine (Tipping, 2001).
• Infinitely many basis functions. The predictive distribution depends on the features only through inner products φ(x)^Tφ(x′). Replacing them with a kernel gives a Gaussian process. This also fixes a weakness visible with localised basis functions: far from every basis centre φ(x*) ≈ 0, so the model variance collapses back to a small value exactly where the model knows least.

Reference: C. M. Bishop, Pattern Recognition and Machine Learning (2006), sections 3.3–3.5, which these formulas and the weight-space and predictive demos follow.