SGD versus Adam Optimization Clarification

In many applications I've seen (e.g. GANs) $\beta_1$ is set to $0$, so $m_1=g_1$, i.e. the numerator of the update rule is the same as in SGD. This leaves two main differences, both related to the MA of the second moment:

1. $v_t:$ raw MA of the second moment serves as a gradient normalizer that divides the gradient by the square root of the moving average of squares of gradients
2. $1-\beta_2$. To reduce bias, $\sqrt{v_t}$ is also divided by $\sqrt{1-\beta_2^t}$. This follows the derivation of the expectation of the square of the gradient, $\mathbf{E}[\big(\frac{\partial E}{\partial w_t}\big)^2]$ in Section 3 of the article. Essentially $\mathbf{E}v_t = (1-\beta^t_2)\mathbf{E}[\big(\frac{\partial E}{\partial w_t}\big)^2] + \varepsilon, $ hence the exppression. Early in the training MAs are close to $0$, and division by $\sqrt{1-\beta_2}$ helps move away from it.

In probability and statistics, moments refer to the uncentered expressions of the form $\mathbf{E}X^k$, which moving averages estimates, hence the name. Normalization allows the gradient adjustment, and, hence, better parameter update