Tune learning rate while tuning other HP

Question

Tune learning rate while tuning other HP

M. Garrigues

2021年10月7日 14:56

When doing hyperparameters optimisation, like a Random Search, should you add a search space for the learning rate ?

My intuition is that some HP might work better with a certain LR, and be sub-optimal with a lower LR. But if I add LR to the search space, I fear that the random search will only favour high LR trials, as they will reach lower loss for the same limited number of max epochs.

What would be the right way to do it ?

Topic hyperparameter-tuning learning-rate machine-learning

Category Data Science

serali · Accepted Answer · 2021年10月7日 14:56

1

serali answered at 2021年10月7日 14:56

Learning rate probably should not be considered an independent hyperparameter as it is usually a good idea to adjust it proportionally to batch size.

Tune learning rate while tuning other HP

About