Golden
LoginSign Up
Highway network

Highway network

Paths along information can flow across several layers optimizing machine learning.

Highway network is a neural network architecture designed to ease gradient-based training of very deep networks. It is built on residual networks (ResNets).

It preserves the shortcuts introduced in the ResNet, but augments them with a adaptable parameter to determine to what extent each layer should be a skip connection or a nonlinear connection. It optimizes networks and increases depth. Highway networks allow training of deep, efficient and multilayered networks.

Its gating mechanisms to regulate information flow across different layers. Highway layers allow the network to copy or transform representations instead of traditional neural layers. A feed forward neural network is defined as,

y = H(x,WH)

while a highway network is defined as,

y = H(x,WH) · T(x,WT) + x · C(x,WC).

T is the transform gate and C is the carry gate, they express how much of the output is produced by transforming the input and carrying it, respectively.

Highway network is used as part of text sequence labeling and speech recognition tasks.

Timeline

People

Name
Role
Related Golden topics

Further reading

Title
Author
Link
Type

Highway Networks

Rupesh Kumar Srivastava, Klaus Greff and Jürgen Schmidhuber

Academic paper

Highway Networks: A new (old) chapter in Deep Learning

Rupesh Kumar Srivastava

ResNets, HighwayNets, and DenseNets, Oh My!

Arthur Juliani

Documentaries, videos and podcasts

Title
Date
Link

Companies

Company
CEO
Location
Products/Services

References