Log in
Enquire now
Vanishing gradient problem

Vanishing gradient problem

The vanishing gradient problem can occur when training neural networks using gradient descent with backpropagation. When the derivative of the activation function tends to be very close to zero, the gradient used to updated the weights of the network may be too small for effective learning.

OverviewStructured DataIssuesContributors

Contents

Other attributes

Wikidata ID
Q18358230

The vanishing gradient problem can occur in artificial neural networks trained using gradient descent with backpropagation. When training such a network, the gradient of the loss function is used to adjust the weights of the network on each iteration. The vanishing gradient problem occurs when the gradient is sufficiently small so as to effectively prevent weights from updating during training, blocking the network from learning.

Networks using activation functions whose derivatives tend to be very close to zero (such as the sigmoid function) are especially susceptible to the vanishing gradient problem. These small values get multiplied at each layer of the network, causing layers closer to the input to adjust their weights very slowly or not at all. Leaky or parametric RELU activation functions provide a robust solution to vanishing gradients by ensuring a derivative sufficiently greater than zero.

Networks using activation functions whose derivatives can take on very large values are susceptible to the related exploding gradient problem.

Timeline

No Timeline data yet.

Further Resources

Title
Author
Link
Type
Date

An Old Problem - Ep. 5 (Deep Learning SIMPLIFIED)

https://www.youtube.com/watch?v=SKMpmAOUa2Q

14 Decmber 2015

On the difficulty of training recurrent neural networks

Razvan Pascanu, Tomas Mikolov and Yoshua Bengio

http://proceedings.mlr.press/v28/pascanu13.pdf

Academic paper

Why are deep neural networks hard to train?

Michael Nielsen

http://neuralnetworksanddeeplearning.com/chap5.html

References

Find more entities like Vanishing gradient problem

Use the Golden Query Tool to find similar entities by any field in the Knowledge Graph, including industry, location, and more.
Open Query Tool
Access by API
Golden Query Tool
Golden logo

Company

  • Home
  • Press & Media
  • Blog
  • Careers
  • WE'RE HIRING

Products

  • Knowledge Graph
  • Query Tool
  • Data Requests
  • Knowledge Storage
  • API
  • Pricing
  • Enterprise
  • ChatGPT Plugin

Legal

  • Terms of Service
  • Enterprise Terms of Service
  • Privacy Policy

Help

  • Help center
  • API Documentation
  • Contact Us
By using this site, you agree to our Terms of Service.