Learning rate terminology, what is 'reducing' a learning rate?
I'm investigating a loss plateau and various techniques for overcoming it, which led me to this page and statement:
Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This callback monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced.
https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ReduceLROnPlateau
I'm confused by this terminology. If my learning rate is 0.001, am I reducing the learning rate when it moves to 0.01, or to 0.0001? I initially would've thought the latter since 0.001 0.0001, but it doesn't make sense to me to change the learning rate to a smaller value when a model has reached a plateau since you would end up making smaller changes to your model than before, making the situation worse.
Topic learning-rate tensorflow machine-learning
Category Data Science