Decreasing Learning Rate doesn't improve the results

In theory and what people are doing (e.g. Paper) decreasing the learning rate should help the optimizer to go deeper into the valley and thus decrease the loss and increase the metric.

Thus, my plan was to train a neural network with a learning rate of 1 until the loss and my metric stay approx. the same for some epochs, then with 0.1, then 0.01 and so on.

However, what I'm observing is, that the loss of the model stagnates after 1000 epochs with learning rate 1 and then decreasing the learning rate multiple times doesn't improve anything.

Do I have to decrease the lr earlier, e.g. after 400 epochs when the neural network is still learning and the loss is going down? Or do you have any ideas why this happens?

Thank you!

Topic loss learning-rate deep-learning machine-learning

Category Data Science

1
answered at

You might want to have a look at several ways of scheduling your learning rate decay, instead of manually trying to optimize it. Check this source from Tensorflow documentation. If you for instance use an exponential learning rate decay schedule, the definition for the learning rate decay would be:

$$learning rate_0 * (decay rate)^\frac{step}{decay steps}$$

but you can try out other schedules too, among which you can find the piecewise constant decay as a simpler one to begin with.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.