A slightly modified version of the machine learning strategy Momentum with stronger theoretical convergence guarantees for convex functions.

Edits on 7 Mar, 2019

Edits made to:
**Description** (+110 characters)
**Article** (+1 images) (+1068 characters)
**Further reading** (+3 rows) (+12 cells) (+489 characters)
**Categories** (+2 topics)
**Related Topics** (+6 topics)

A slightly modified version of Momentum with stronger theoretical convergence guarantees for convex functions.

Article

Nesterov momentum, or Nesterov Accelerated Gradient (NAG), is a slightly modified version of Momentum with stronger theoretical convergence guarantees for convex functions. In practice, it has produced slightly better results than classical Momentum.

Difference Between Momentum and Nesterov Momentum

In the standard Momentum method, the gradient is computed using current parameters (* θt*). Nesterov momentum achieves stronger convergence by applying the velocity (

The reason this is sometimes referred to as a "lookahead" gradient is that computing the gradient based on interim parameters allow NAG to change velocity in a faster and more responsive way, resulting in more stable behavior than classical Momentum in many situations, particularly for higher values of *μ. *NAG is the correction factor for classical Momentum method.

Further reading

Title

Author

Link

Type

CS231n Convolutional Neural Networks for Visual Recognition

Stanford Computer Science

Web

Momentum Method and Nesterov Accelerated Gradient - Konvergen - Medium

Roan Gylberth

Web

On the importance of initialization and momentum in deep learning

Ilya Sutskever, James Martens, George Dahl, Geoffrey Hinton

Related Topics

Edits made to:

A slightly modified version of the machine learning strategy Momentum with stronger theoretical convergence guarantees for convex functions.

Text is available under the Creative Commons Attribution-ShareAlike 4.0; additional terms apply. By using this site, you agree to our Terms & Conditions.