LoginSign Up
Exploding gradient problem

Exploding gradient problem

The exploding gradient problem is a difficulty which can occur when training artificial neural networks using gradient descent by backpropagation. When large error gradients accumulate the model may become unstable and impair effective learning.

The exploding gradient problem is a difficulty which can occur when training artificial neural networks using gradient descent by backpropagation. When large error gradients accumulate the model may become unstable and impair effective learning.

Change in model weights can create an unstable network. The values of weights can become so large and cause overflow. A gradient is the direction and magnitude calculated during the training of a neural network it is used to teach the network weights in the right direction by the right amount. When there is an error gradient, explosion of components may grow exponentially.

Exploding gradient problem can be addressed by redesigning the network model, using rectified linear activation, using long short term memory (LSTM) networks, gradient clipping and weight regularization.

Timeline

People

Name
Role
Related Golden topics

Further reading

Title
Author
Link
Type

On the difficulty of training recurrent neural networks

Razvan Pascanu, Tomas Mikolov and Yoshua Bengio

Academic paper

Understanding the exploding gradient problem

Razvan Pascanu, Tomas Mikolov and Yoshua Bengio

Academic paper

Documentaries, videos and podcasts

Title
Date
Link

Companies

Company
CEO
Location
Products/Services