Contents

# Ridge Regression

#### What is Ridge Regression?

Ridge regression is a method used to solve the least-squares problem (minimizing the sum of squares) by adding regularization. As a modeling tool, it is applied when the number of features in a set exceeds the number of elements in a dataset or when the dataset has multicollinearity. Its name derives from the fact that the diagonal of ones in the correlation matrix can be described as a ridge.

This method is a special case of Tikhonov regularization but applies to smaller sets. The Tikhonov method is also used when a dataset contains a high degree of noise.

Mathematically, the Ridge regression can be understood as an extension of Ordinary Least Squares (OLS) where a parameter k (called ridge parameter) is added to the cross product matrix.

This extension can be seen by using matrix algebra:

given a dataset, and an equation defined as:

where Y represents the independent variable, X the dependent variable, and B the coefficients of the estimated equation, OLS estimates the coefficients as:

where Ridge regression adds regularization as:

with k≥0.

The idea behind ridge regression is to use a shrinkage estimator (k) that shrinks the solution to values closer to the “true” population parameters. However, finding the right k value is not straightforward. This is the reason why ridge regression is not as frequently used as OLS.

#### Why is Ridge Regression Important?

Ridge regression is important in the presence of multicollinearity, which occurs when the features defining a dataset are not completely independent of each other. Besides, it is also used when the number of data rows is small compared to the number of features. As a rule of thumb, this method is useful when the number of data points is less than one hundred thousand.

#### Ridge Regression and LogicPlum

Although calculating the ridge coefficient may require serious computation, LogicPlum’s platform relieves the user from this task. Its automated mechanism allows experts and non-experts alike to model phenomena without the need to deepen into the necessary mathematical and statistical knowledge.

##### Additional Resources

Wikipedia. (2020). *Tikhonov Regularization. *Available at https://en.wikipedia.org/wiki/Tikhonov_regularization