Regularizing the intercept - particular case

Yesterday I posted this thread Regularizing the intercept where I had a question about penalizing the intercept. In short, I asked wether there exist cases where penalizing the intercept leads to a lower expected prediction error and the answer was:

Of course there exist scenarios where it makes sense to penalize the intercept, if that aligns with domain knowledge. However in real world, more often we do not just penalize the magnitude of intercept, but enforce it to be zero. This happens in cases where we expect the output to be 0 if all inputs are 0.

Now my teacher simply states that in all cases the model will get a lower Expected Prediction Error if we do not penalize the intercept. Thus, I am trying to construct a specific case where penalizing the intercept leads to a lower or unchanged EPE compared to not penalizing the intercept, but so far I have been unsuccesful. Can anyone help me construct a case where we use ridge or lasso regularization on all there features (including the intercept) and thereby get a lower EPE that we would have if the intercept was not penalized? Or confirm that my teacher is right in the statement that we always get a lower EPE if we do not penalize the intercept?

EDIT In The Elements of Statistical Learning, regarding regularized logistic regression, it says:

As with the lasso, we typically do not penalize the intercept term

The edit is there to point out that it might be through logistic regression that a case could be made

Thanks!

Topic lasso ridge-regression regularization

Category Data Science

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.