From Neuron to Neural Network
Each neuron takes m Inputs x and has m Weights (calculated during AI model training) multiple them and apply an Activation function to produce an Output. Weight 0 is called and a bias.
Neural Network consists of multiple layers of Neurons stacked one after another. First Layer is called input layer and the last layer is output layer and layers in between are hidden layers.
Types of Neural Network
Vector to Sequence model —It takes a vector and produce a sequence. e.g. take a picture and produce list of words describing the picture.( a dog picture may produce “A large dog” output)
Sequence to Vector model — Takes a Sequence and produces an Vector. Use case — sentiment analysis.
Sequence to Sequence model — Takes a Sequence and produces a Sequence. Use Case — auto complete sentences, predicting the next word during typing
Encoder Decoder Model — output from Sequence to Vector model is fed into Vector to Sequence model. Use Case — language translation
Addressing Vanishing and Exploding gradient
1- Introduce skip connections ( ResNet architecture)
2- Remove length 1 connections and replace them with long connection
3- Introduce Leaky Recurrent unit.
4- Gated Recurrent Networks. Introduces variables to train to make network remember or forget. (LSTM)
