Neural Networks from scratch

Introduction to Neural Networks

A neural network is a network which closely simulates the learning of a human brain. This is done by using connection weights and biases for each neuron in every layer.

A depiction of how the information passes through a neural network

Structure of a neural network

A neural network has many layers with different or same number of neurons and hence deciding the structure of a neural network can be handy.

Activation functions

Just calculating the weighted sum is not enough for a neural network to learn and hence we need to use some kind of non linear function which transforms the input so that it can be fed in the next layer. For this purpose, we have many activation functions

Sigmoid activation function
  1. The value of function at 0 is 0.5 , so 0.5 can be used as a classification threshold for two class classification, where the inputs whose sigmoid value is less than 0.5 can be classified as one category and the inputs whose sigmoid value greater than 0.5 can be classified as another category
tanh activation function
  1. It is centered at 0 as the value at x=0 is 0
  1. This is centered around 0 which further makes the calculations easier
  2. It standardizes the inputs and converts them from (-1,1) with mean = 0
Rectified Linear Unit function
  1. The slope of the output for all positive inputs is 1
Leaky Rectified Linear Unit
  1. The slope of the output for all positive inputs is 1
Softmax activation function
  1. The outputs of the softmax function sum up to 1

Forward Propagation in Neural Network

Forward propagation through a neural network can be considered as

Equation of forward propagation in a neural network
Forward propagation of a 3 layered neural network

Back Propagation

Back Propagation through a neural network

Training the neural network

In order to train a neural network, first we need to perform forward propagation, then we need to calculate the changes in weights and biases for the neural network and then we need to update the weights and biases in order to improve them. We usually perform the training for many iterations in order to obtain the best possible parameters

Forward propagation in a neural network
Backward propagation in a 4 layered neural network
Simplified backward propagation of neural network
Updating weights

--

--

A data science enthusiast currently pursuing a bachelor's degree in data science

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aayushmaan Jain

A data science enthusiast currently pursuing a bachelor's degree in data science