The following question is based purely on the material available on MIT's open courseware youtube channel. (https://www.youtube.com/watch?v=wrEcHhoJxjM). In it, Professor Gilbert Strang explains the general formulation of the momentum gradient descent problem and ultimately arrives at optimum values (40:05 in the video) for the variables $s$ and $\beta$. \ $\textbf{Background}$ Lets begin with the standard gradient descent not covered in this video. The equation for this is: $x_{k+1}=x{k}-s \nabla f(x_k) $ $s$ is the step size, $f(x_k)$ is the value …

While using "Two class neural network" in Azure ML, I encountered "Momentum" property. As per documentation, which is not clear, it says For The momentum, type a value to apply during learning as a weight on nodes from previous iterations. Although that is not very clear. Can someone please explain?

The Adam optimizer is often used for training neural networks; it typically avoids the need for hyperparameter search over parameters like the learning rate, etc. The Adam optimizer is an improvement on gradient descent. I have a situation where I want to use projected gradient descent (see also here). Basically, instead of trying to minimize a function $f(x)$, I want to minimize $f(x)$ subject to the requirement that $x \ge 0$. Projected gradient descent works by clipping the value of …

The "momentum" adds a little of the history of the last weight updates to the actual update, with diminishing weight history (older momentum shares get smaller). Is it significiantly superior? Weightupdate: $$ w_{i+1} = w_i + m_i $$ With momentum: $$ m_0 = 0 \\ m_1 = \Delta w_{1} + \beta m_0 = \Delta w_1 \\ m_2 = \Delta w_{2} + \beta m_1 = \Delta w_2 + \beta\Delta w_1 $$ So the momentum already contains the actual weightupdate and the …

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (vanilla gradient descent did). Why might this be? Later steps where affected by momentum etc but I would assume these effects wouldn’t come into play for the first few steps.

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space v_t \space \text{is second moment of gradient} $$ I have the following questions with regards to the above formula: What exactly is first and second moments of gradients ? what's the intuition behind the first and second moment's formula? I understand SGD with momentum and SGD with RMSprop but here we are …

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD makes a huge difference in the optimization process. When I use vanilla SGD, optimization is really smooth. Training and validation decrease at a similar rate, and seem to converge properly past a sufficiently large number of epochs: However, when I switch to NAG, without changing any other parameters, suddenly the validation …

If momentum optimizer independently keeps a custom "inertia" value for each weight, then why do we ever need to bother with learning rate? Surely, momentum would catch up its magnutude pretty quickly to any needed value anyway, why to bother scaling it with learning rate? $$v_{dw} = \beta v_{dw} +(1-\beta)dW$$ $$W = W-\alpha v_{dw}$$ Where $\alpha$ is the learning rate (0.01 etc) and $\beta$ is the momentum coefficient (0.9 etc) Edit Thanks for the answer! To put it more plain: …