LoginSign Up
Exploding gradient problem

Exploding gradient problem

The exploding gradient problem is a difficulty which can occur when training artificial neural networks using gradient descent by backpropagation. When large error gradients accumulate the model may become unstable and impair effective learning.

The exploding gradient problem is a difficulty which can occur when training artificial neural networks using gradient descent by backpropagation. When large error gradients accumulate the model may become unstable and impair effective learning.

Change in model weights can create an unstable network. The values of weights can become so large and cause overflow. A gradient is the direction and magnitude calculated during the training of a neural network it is used to teach the network weights in the right direction by the right amount. When there is an error gradient, explosion of components may grow exponentially.

Exploding gradient problem can be addressed by redesigning the network model, using rectified linear activation, using long short term memory (LSTM) networks, gradient clipping and weight regularization.

Timeline

Currently, no events have been added to this timeline yet.
Be the first one to add some.

People

Name
Role
Related Golden topics

Further reading

Author
Title
Link
Type

Razvan Pascanu, Tomas Mikolov and Yoshua Bengio

On the difficulty of training recurrent neural networks

Academic paper

Razvan Pascanu, Tomas Mikolov and Yoshua Bengio

Understanding the exploding gradient problem

Academic paper

Documentaries, videos and podcasts

Title
Date
Link

Companies

Company
CEO
Location
Products/Services