Equivalence of Bayesian priors and Ridge regression

Today I’m going to take you through the comparison of a Bayesian formalism for regression and compare it to Ridge regression which is a penalized version of OLS. The rationale I have for doing so is that many times in my career I’ve come across “frequentists” who claim that parameters can be controlled via a process called shrinkage, regularization, weight decay, or weight elimination depending on whether you’re using GLM’s, SVM’s or Neural networks. This statement is in principle correct while misguided. The regularization can be seen to arise as a consequence of a probabilistic formulation. I would go so far as to say that there is no such thing as frequentist statistics; there are only those who refuse to add prior information to their model! Before we get started I would like to warn you that this post is going to get a tad mathematical. If that scares you, you might consider skipping the majority of this post and go directly to the summary. Now, let’s go!

In this view it is clear that we want to learn something about since that’s the unknowns. The other parts we have observed data on. So we would like to say something clever about p(β| y, X). How do we go about doing that? Well for starters we need to realize that p(y, X, β) can actually be written as

which means that

and therefor

which is just a derivation of Baye’s rule. Now we actually have something a bit more useful at our hands which is ready to be interpreted and implemented. What do I mean by implemented? Seems like an odd thing to say about probability distributions right? As weird as it may seem we actually haven’t given the probability distributions a concise mathematical representation. This is of course necessary for any kind of inference. So let’s get to it. The first term I would like to describe is the likelihood i.e. the p(y|β, X) which describes the likelihood of observing the data given the covariance matrix and a set of parameters β. For simplicity let’s say this probability distribution is gaussian thus taking the following form p(y|β, X) = N(y-βX; 0,σ). This corresponds to setting up a measurement model yt=βxt + ????where ???? ~ N(0,σ).

The second term in the nominator on the right hand side is our prior p(β) which we will also consider gaussian. Thus, we will set p(β)=N(0,αI) indicating that the parameters are independant from each other and most likely centered around with 0 a known standard deviation of α. The last term is the denominator p(y) which in this setting functions as the evidence. This is also the normalizing constant that makes sure that we can interpret the right hand side probabilistically.

That’s it! We now have the pieces we need to push the inference button. This is often for more complicated models done by utilizing Markov Chain Monte Carlo methods to sample the distributions. If we are not interested in the distribution but only the average estimates for the parameters we can just turn this into an optimization problem instead by realizing that

since p(y) just functions as a normalizing constant and doesn’t change the location of the β that would yield the maximum probability. Thus we can set up the optimization problem as

and maximize this function. Normally when we solve optimization problems it’s easier and nicer to turn it into a minimization problem instead of a maximization problem. This is easily done by minimizing

as opposed to the equation before. For the sake of clarity let’s assume from now on that we only have one independent variable and only one parameter β. Since we know that

we can easily unfold the logarithms to reveal

which can be more nicely written as

where C=TC1-C2. As such putting a Gaussian prior on your β is equivalent to penalizing solutions that differ from 0 by a factor of 1/(2σ^2) i.e. 1 divided by two times the variance of the Gaussian. Thus, the smaller the variance, the higher our prior confidence is that the solution should be close to zero. The larger the variance the more uncertain we are about where the solution should end up.

where the variables and parameters have the same interpretation as before. This is basically Ordinary Least Squares (OLS) which suffers from overfitting and sensitivity to outliers and multicollinearity. So what Ridge regression does is to introduce a penalty term to this set of equations like this

where ???? is typically chosen to be γ????. This means that all values in the parameter vector β should be close to 0. Continuing along this track we can select a dumbed down version of this equation to show what’s going on for a simple application of one variable x and one parameter β. In this case

turns into

which you may recognize from before. Not convinced? Well let’s look into the differences.

Here it’s pretty obvious to see that they are equivalent. The constant C plays no role in the minimization of these expressions. Neither does the denominator 2σ^2. Thus if we set

the equivalence is clear and we see that what we are really minimizing is

which concludes my point.