Deep Learning Assignment 1 Solution

Name: Vivek Rana

Roll No.: 1709113908

1. Explain the following:

i. Perceptrons:
Ans: A perceptron is a neural network unit (an artificial neuron) that does certain computations to
detect features or business intelligence by learning the weights for the input signals in order to draw a
linear decision boundary.

A Perceptron can also be said as an algorithm for supervised learning of binary classifiers. This
algorithm enables neurons to learn and processes elements in the training set one at a time.
There are two types of Perceptrons: Single layer Perceptrons, learns only linearly separable patterns
while Multilayer Perceptrons or feedforward neural networks, containing two or more layers with
greater processing power.

ii. Logistic Regression:

Ans: Logistic regression is a statistical model that in its basic form uses a logistic function to model a
binary dependent variable. In regression analysis, logistic regression (or logit regression) is estimating
the parameters of a logistic model (a form of binary regression)

2. Explain the statement “ Neural Network is a function approximator”.

Ans: Supervised learning in machine learning can be described in terms of function approximation.
Neural networks are an example of a supervised machine learning algorithm that is perhaps best
understood in the context of function approximation. This can be demonstrated with examples of
neural networks approximating simple one-dimensional functions that aid in developing the intuition
for what is being learned by the model.
 3. What is a Neural Network? What are its advantages?

Ans: A neural network is a network or circuit of neurons, or in a modern sense, an artificial neural
network, composed of artificial neurons or nodes. Artificial neural networks or connectionist systems
are computing systems vaguely inspired by the biological neural networks that constitute animal brains.
Such systems "learn" to perform tasks by considering examples, generally without being programmed
with task-specific rules

Advantages of Neural Networks ( ANN):

i. Storing information on the entire network

ii. Ability to work with incomplete knowledge
iii. Having fault tolerance
iv. Having a distributed memory
v. Gradual corruption
vi. Ability to make machine learning
vii. Parallel processing capability

4. Difference between Machine Learning and Deep Learning.

Ans: Factors determining the differences between ML and DL are:

i. Data dependencies: The most important difference between deep learning and traditional machine
learning is its performance as the scale of data increases. When the data is small, deep learning
algorithms don’t perform that well. This is because deep learning algorithms need a large amount of
data to understand it perfectly. On the other hand, traditional machine learning algorithms with their
handcrafted rules prevail in this scenario. Below image summarizes this fact.

ii. Hardware dependencies: Deep learning algorithms heavily depend on high-end machines, contrary
to traditional machine learning algorithms, which can work on low-end machines. This is because the
requirements of deep learning algorithm include GPUs which are an integral part of its working.

iii. Feature engineering: Feature engineering is a process of putting domain knowledge into the
creation of feature extractors to reduce the complexity of the data and make patterns
more visible to learning algorithms to work. This process is difficult and expensive in terms of time and
expertise.

In Machine learning, most of the applied features need to be identified by an expert and then hand-
coded as per the domain and data type.

Deep learning algorithms try to learn high-level features from data. This is a very distinctive part of
Deep Learning and a major step ahead of traditional Machine Learning. Therefore, deep learning
reduces the task of developing new feature extractor for every problem.

iv. Problem Solving Approach: When solving a problem using traditional machine learning algorithm, it
is generally recommended to break the problem down into different parts, solve them individually and
combine them to get the result. Deep learning in contrast advocates to solve the problem end-to-end.

v. Execution time: In Deep Learning, mathematically you can find out which nodes of a deep neural
network were activated, but we don’t know what there neurons were supposed to model and what
these layers of neurons were doing collectively. So we fail to interpret the results. On the other hand,
machine learning algorithms like decision trees give us crisp rules as to why it chose what it chose, so it
is particularly easy to interpret the reasoning behind it.

5. List some disadvantages of Neural Networks.

i. Hardware dependence: Artificial neural networks require processors with parallel processing
power, in accordance with their structure. For this reason, the realization of the equipment is
dependent.
ii. Unexplained behavior of the network: This is the most important problem of ANN. When
ANN produces a probing solution, it does not give a clue as to why and how. This reduces trust in
the network.
iii. Determination of proper network structure: There is no specific rule for determining the structure
of artificial neural networks. Appropriate network structure is achieved through experience and
trial and error.
iv. Difficulty of showing the problem to the network: ANNs can work with numerical information.
Problems have to be translated into numerical values before being introduced to ANN. The display
mechanism to be determined here will directly influence the performance of the network . This
depends on the user's ability.
v. The duration of the network is unknown

6. Explain Stochastic Gradient Descent.

The word ‘stochastic‘ means a system or a process that is linked with a random probability. Hence, in
Stochastic Gradient Descent, a few samples are selected randomly instead of the whole data set for
each iteration. In Gradient Descent, there is a term called “batch” which
 denotes the total number of samples from a dataset that is used for calculating the gradient for each
iteration. In typical Gradient Descent optimization, like Batch Gradient Descent, the batch is taken to
be the whole dataset. Although, using the whole dataset is really useful for getting to the minima in a
less noisy or less random manner, but the problem arises when our datasets get really huge.

7. Explain the following Loss functions:

i. Mean Squared Error: In statistics, the mean squared error or mean squared deviation of an
estimator measures the average of the squares of the errors—that is, the average squared
difference between the estimated values and the actual value. MSE is a risk function,
corresponding to the expected value of the squared error loss
ii. Cross-Entropy: Cross-entropy loss, or log loss, measures the performance of a classification model
whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted
probability diverges from the actual label.

8. What is the difference between backpropagation and the forward pass? Ans:

Forward propagation:

In forward Propagation we provides input x each nueron wull calculate two functions

one is linear multiplication i.e Z= W*X+b and we use activation function a=relu(z) (use different
activation functions) then it will forward through every layer and we will get predicted output.

Back propagation:

Back propagation is a technique to reduce the loss i.e.( Actual o/p-predicted o/p) by updating the
parameters weight, bias by using an algorithm called Gradient descent. So technically both are
different in back propagation we use Gradient Descent algorithm. This is a credit assignment property
that means the reason for wrong output not only due to final layer but also due to previous layer,That’s
why we calculate gradients of every layer w.r.t Loss(L)

9. Derive the weight updation formula for a neural network with one input layer, one hidden layer
and one output layer each having the following nodes respectively: 3,2,1. [There is no activation
layer]
Ans:
Forward Pass:
Inputs: Node 1: x1, Node 2: x2, Node 3: x3

Hidden Layer:
Node 4: W1*[ x1 x2 x3 ] + b1 = Y1
Node 5: W2 *[ x1 x2 x3 ] + b2 = Y2
 Output Layer:
Node 6: W3*[ Y1 Y2 ] + b3 = YActual

Error Function:
Mean Square Error:
ΔE = 1/2 *( YActual -YPred )2

Backpropagation: Weights
updatation:
At Output Layer:
W3 = W3 + W3(ΔE)
At Hidden Layers:
Wi = Wi + α*(ΔE)*xi

10. Which machine learning algorithm uses a maximum margin techniqu? Explain it in detail.
Ans: In machine learning, a margin classifier is a classifier which is able to give an associated distance
from the decision boundary for each example. For instance, if a linear classifier (e.g. perceptron or
linear discriminant analysis) is used, the distance (typically euclidean distance, though others may
be used) of an example from the separating hyperplane is the margin of that example.
The notion of margin is important in several machine learning classification algorithms, as it can be
used to bound the generalization error of the classifier. These bounds are frequently shown using the
VC dimension. Of particular prominence is the generalization error bound on boosting algorithms
and support vector machines.