Why is the regularization parameter not applied to the intercept parameter?
From what I have read about the cost functions for Linear and Logistic regression, the regularization parameter (λ) is applied to all terms except the intercept. For example, here are the cost functions for linear and logistic regression respectively (Notice that j starts from 1):
Regularization tries to reduce the chance of overfitting by reducing the sensitivity to small changes in the input data. This is not as much of an issue for the intercept term (relative to the coefficients) so it is often not included.
The main aim of a regularization is to reduce the parameters values which in turn will reduce the effect of the feature associated with that parameter on the models prediction. For example, Since the intercept Theta0 basically doesn't have a feature associated with it, We don't penalize it.
Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.