Ridge Regression

What is Ridge Regression?

Ridge regression is a method used to solve the least-squares problem (minimizing the sum of squares) by adding regularization. As a modeling tool, it is applied when the number of features in a set exceeds the number of elements in a dataset or when the dataset has multicollinearity. Its name derives from the fact that the diagonal of ones in the correlation matrix can be described as a ridge.

This method is a special case of Tikhonov regularization but applies to smaller sets. The Tikhonov method is also used when a dataset contains a high degree of noise.

Mathematically, the Ridge regression can be understood as an extension of Ordinary Least Squares (OLS) where a parameter k (called ridge parameter) is added to the cross product matrix.

This extension can be seen by using matrix algebra:

given a dataset, and an equation defined as:

where Y represents the independent variable, X the dependent variable, and B the coefficients of the estimated equation, OLS estimates the coefficients as:

where Ridge regression adds regularization as:

with k≥0.

The idea behind ridge regression is to use a shrinkage estimator (k) that shrinks the solution to values closer to the “true” population parameters. However, finding the right k value is not straightforward. This is the reason why ridge regression is not as frequently used as OLS.

Why is Ridge Regression Important?

Ridge regression is important in the presence of multicollinearity, which occurs when the features defining a dataset are not completely independent of each other. Besides, it is also used when the number of data rows is small compared to the number of features. As a rule of thumb, this method is useful when the number of data points is less than one hundred thousand.

Ridge Regression and LogicPlum

Although calculating the ridge coefficient may require serious computation, LogicPlum’s platform relieves the user from this task. Its automated mechanism allows experts and non-experts alike to model phenomena without the need to deepen into the necessary mathematical and statistical knowledge.

Additional Resources

Wikipedia. (2020). Tikhonov Regularization. Available at https://en.wikipedia.org/wiki/Tikhonov_regularization