Received April 1, 2019, accepted April 15, 2019, date of publication April 22, 2019, date of current version

May 1, 2019.
Digital Object Identifier 10.1109/ACCESS.2019.2912200

Review of Deep Learning Algorithms

and Architectures
AJAY SHRESTHA AND AUSIF MAHMOOD, (Senior Member, IEEE)
Department of Computer Science and Engineering, University of Bridgeport, Bridgeport, CT 06604, USA
Corresponding author: Ajay Shrestha (shrestha@my.bridgeport.edu)

ABSTRACT Deep learning (DL) is playing an increasingly important role in our lives. It has already made a
huge impact in areas, such as cancer diagnosis, precision medicine, self-driving cars, predictive forecasting,
and speech recognition. The painstakingly handcrafted feature extractors used in traditional learning,
classification, and pattern recognition systems are not scalable for large-sized data sets. In many cases,
depending on the problem complexity, DL can also overcome the limitations of earlier shallow networks
that prevented efficient training and abstractions of hierarchical representations of multi-dimensional training
data. Deep neural network (DNN) uses multiple (deep) layers of units with highly optimized algorithms and
architectures. This paper reviews several optimization methods to improve the accuracy of the training and
to reduce training time. We delve into the math behind training algorithms used in recent deep networks.
We describe current shortcomings, enhancements, and implementations. The review also covers different
types of deep architectures, such as deep convolution networks, deep residual networks, recurrent neural
networks, reinforcement learning, variational autoencoders, and others.

INDEX TERMS Machine learning algorithm, optimization, artificial intelligence, deep neural network
architectures, convolution neural network, backpropagation, supervised and unsupervised learning.

I. INTRODUCTION Neural Networks can be used in a variety of prob-

Neural Network is a machine learning (ML) technique that is lems including pattern recognition, classification, clustering,
inspired by and resembles the human nervous system and the dimensionality reduction, computer vision, natural language
structure of the brain. It consists of processing units organized processing (NLP), regression, predictive analysis, etc. Here
in input, hidden and output layers. The nodes or units in is an example of image recognition.
each layer are connected to nodes in adjacent layers. Each Figure 1 shows how a deep neural network called Convo-
connection has a weight value. The inputs are multiplied lution Neural Network (CNN) can learn hierarchical levels
by the respective weights and summed at each unit. The of representations from a low-level input vector and success-
sum then undergoes a transformation based on the activa- fully identify the higher-level object. The red squares in the
tion function, which is in most cases is a sigmoid function, figure are simply a gross generalization of the pixel values of
tan hyperbolic or rectified linear unit (ReLU). These func- the highlighted section of the figure. CNNs can progressively
tions are used because they have a mathematically favorable extract higher representations of the image after each layer
derivative, making it easier to compute partial derivatives of and finally recognize the image.
the error delta with respect to individual weights. Sigmoid The implementation of neural networks consists of the
and tanh functions also squash the input into a narrow output following steps:
range or option, i.e., 0/1 and −1/+1 respectively. They imple- 1. Acquire training and testing data set
ment saturated nonlinearity as the outputs plateaus or satu- 2. Train the network
rates before/after respective thresholds. ReLu on the other 3. Make prediction with test data
hand exhibits both saturating and non-saturating behaviors
The paper is organized in the following sections:
with f (x) = max(0, x). The output of the function is then fed
as input to the subsequent unit in the next layer. The result of 1. Introduction to Machine Learning
the final output layer is used as the solution for the problem. a. Background and Motivation
2. Classifications of Neural Networks
The associate editor coordinating the review of this manuscript and 3. DNN Architectures
approving it for publication was Geng-Ming Jiang. 4. Training Algorithms
2169-3536
2019 IEEE. Translations and content mining are permitted for academic research only.
53040 Personal use is also permitted, but republication/redistribution requires IEEE permission. VOLUME 7, 2019
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

and application of backpropagation algorithm. The automa-

tion of feature extractors is what differentiates a DNNs from
earlier generation machine learning techniques.
DNN is a type of neural network modeled as a multilayer
perceptron (MLP) that is trained with algorithms to learn
representations from data sets without any manual design
of feature extractors. As the name Deep Learning suggests,
it consists of higher or deeper number of processing lay-
ers, which contrasts with shallow learning model with fewer
layers of units. The shift from shallow to deep learning has
allowed for more complex and non-linear functions to be
mapped, as they cannot be efficiently mapped with shallow
architectures. This improvement has been complemented by
the proliferation of cheaper processing units such as the
general-purpose graphic processing unit (GPGPU) and large
volume of data set (big data) to train from. While GPGPUs are
less powerful that CPUs, the number of parallel processing
cores in them outnumber CPU cores by orders of magnitude.
This makes GPGPUs better for implementing DNNs. In addi-
tion to the backpropagation algorithm and GPU, the adoption
and advancement of ML and particularly Deep Learning can
FIGURE 1. Image recognition by a CNN.
be attributed to the explosion of data or bigdata in the last
10 years. ML will continue to impact and disrupt all areas
5. Shortcomings of Training Algorithms of our lives from education, finance, governance, healthcare,
6. Optimization of Training Algorithms manufacturing, marketing and others [7].
7. Architectures & Algorithms – Implementations
8. Conclusion B. MOTIVATION
Deep learning is perhaps the most significant development in
A. BACKGROUND the field of computer science in recent times. Its impact has
In 1957, Frank Rosenblatt created the perceptron, the first been felt in nearly all scientific fields. It is already disrupting
prototype of what we now know as a neural network [1]. and transforming businesses and industries. There is a race
It had two layers of processing units that could recognize among the world’s leading economies and technology compa-
simple patterns. Instead of undergoing more research and nies to advance deep learning. There are already many areas
development, neural networks entered a dark phase of its where deep learning has exceeded human level capability
history in 1969, when professors at MIT demonstrated that and performance, e.g., predicting movie ratings, decision to
it couldn’t even learn a simple XOR function [2]. approve loan applications, time taken by car delivery, etc. [8].
In addition, there was another finding that particularly On March 27, 2019 the three deep learning pioneers (Yoshua
dampened the motivation for DNN. The universal approxi- Bengio, Geoffrey Hinton, and Yann LeCun) were awarded
mation theorem showed that a single hidden layer was able the Turing Award, which is also referred to as the ‘‘Nobel
to solve any continuous problem [3]. It was mathematically Prize’’ of computing[9]. While a lot has been accomplished,
proven as well [4], which further questioned the validity there is more to advance in deep learning. Deep learning
of DNN. While a single hidden layer could be used to learn, has a potential to improve human lives with more accurate
it was not efficient and was a far cry from the convenience and diagnosis of diseases like cancer [10], discovery of new drugs,
capability afforded by the hierarchical abstraction of multiple prediction of natural disasters [11]. E.g., [12] reported that an
hidden layers of DNN that we know now. But it was not deep learning network was able to learn from 129,450 images
just the universal approximation theorem that held back the of 2,032 diseases and was able to diagnose at the same level
progress of DNN. Back then, we didn’t have a way to train a as 21 board certified dermatologists. Google AI [10] was able
DNN either. These factors prolonged the so-called AI winter, to beat the average accuracy of US board certified general
i.e., a phase in the history of artificial intelligence where it pathologists in grading prostate cancer by 70% to 61%.
didn’t get much funding and interest, and as a result didn’t The goal of this review is to cover the vast subject of deep
advance much either. learning and present a holistic survey of dispersed informa-
A breakthrough in DNN occurred with the advent of tion under one article. It presents novel work by collating the
backpropagation learning algorithm. It was proposed in works of leading authors from the wide scope and breadth
the 1970s [5] but it wasn’t until mid-1980s [6] that it was fully the deep learning. Other review papers [13]–[16] focus on
understood and applied to neural networks. The self-directed specific areas and implementations without encompass the
learning was made possible with the deeper understanding full scope of the field. This review covers the different types

VOLUME 7, 2019 53041

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

the subscript letters and b represents the bias value of the

unit.
Unlike feedforward neural networks, the processing units
in RNN form a cycle. The output of a layer becomes the
input to the next layer, which is typically the only layer in the
network, thus the output of the layer becomes an input to itself
forming a feedback loop. This allows the network to have
memory about the previous states and use that to influence
the current output. One significant outcome of this difference
is that unlike feedforward neural network, RNN can take a
sequence of inputs and generate a sequence of output values
as well, rendering it very useful for applications that require
processing sequence of time phased input data like speech
recognition, frame-by-frame video classification, etc.
Figure 2b demonstrates the unrolling of a RNN in time.
E.g., if a sequence of 3-word sentence constitutes an input,
then each word would correspond to a layer and thus the
network would be unfolded or unrolled 3 times into a
3-layer RNN.
Here is the mathematical explanation of the diagram:
xt represents the input at time t. U , V , and W are the
learned parameters that are shared by all steps. Ot is the
output at time t. St represents the state at time t and can
be computed as follows, where f is the activation function,
e.g., ReLU.
St = f (Uxt + Wst−1 ) (1)
Radial basis function neural network is used in classifi-
cation, function approximation, time series prediction prob-
lems, etc. It consists of input, hidden and output layers. The
hidden layer includes a radial basis function (implemented as
FIGURE 2. (a) Feedforward neural network [6]. (b) The unrolling of RNN
gaussian function) and each node represents a cluster center.
in time [6]. The network learns to designate the input to a center and
the output layer combines the outputs of the radial basis
function and weight parameters to perform classification or
of deep learning network architectures, deep learning algo-
inference [17].
rithms, their shortcomings, optimization methods and the
Kohonen self-organizing neural network self organizes the
latest implementations and applications.
network model into the input data using unsupervised learn-
ing. It consists of two fully connected layers, i.e., input layer
II. CLASSIFICATION OF NEURAL NETWORK and output layer. The output layer is organized as a two-
Neural Networks can be classified into the following different dimensional grid. There is no activation function and the
types. weights represent the attributes (position) of the output layer
1. Feedforward Neural Network node. The Euclidian distance between the input data and each
2. Recurrent Neural Network (RNN) output layer node with respect to the weights are calculated.
3. Radial Basis Function Neural Network The weights of the closest node and its neighbors from the
4. Kohonen Self Organizing Neural Network input data are updated to bring them closer to the input data
5. Modular Neural Network with the formula below [18].
In feedforward neural network, information flows in
wi (t + 1) = wi (t) + α(t)ηj∗i (x(t) − wi (t)) (2)
just one direction from input to output layer (via hidden
nodes if any). They do not form any circles or loopbacks. where x(t) is the input data at time t, wi (t) is the ith weight at
Figure 2a shows a particular type of implementation of a time t and ηj∗i is the neighborhood function between the ith
multilayer feedforward neural network with values and func- and jth nodes.
tions computed along the forward pass path. Z is the weighed Modular neural network breaks down large network into
sum of the inputs and y represents the non-linear activation smaller independent neural network modules. The smaller
function f of Z at each layer. W represents the weights networks perform specific task which are later combined as
between the two units in the adjoining layers indicated by part of a single output of the entire network [19].

53042 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

DNNs are implemented in the following popular ways: for camera motion estimation and monocular depth. Modi-
1. Sparse Autoencoders fied Neural Networks such as Deep Belief Network (DBM)
2. Convolution Neural Networks (CNNs or ConvNets) as described by Chen and Lin [23] uses both labeled and
3. Restricted Boltzmann Machines (RBMs) unlabeled data with supervised and unsupervised learning
4. Long Short-Term Memory (LSTM) respectively to improve performance. Developing a way
Autoencoders are neural networks that learn fea- to automatically extract meaningful features from labeled
tures or encoding from a given dataset in order to perform and unlabeled high dimensional data space is challenging.
dimensionality reduction. Sparse Autoencoder is a variation Yann LeCun et al. asserts that one way we could achieve this
of Autoencoders, where some of the units output a value would be to utilize and integrate both unsupervised and super-
close to zero or are inactive and do not fire. Deep CNN vised learning [24]. Complementing unsupervised learning
uses multiple layers of unit collections that interact with (with un-labeled data) with supervised learning (with labeled
the input (pixel values in the case of image) and result in data) is referred to as semi-supervised learning.
desired feature extraction. CNN finds it application in image DNN and training algorithms have to overcome two major
recognition, recommender systems and NLP. RBM is used to challenges: premature convergence and overfitting. Prema-
learn probability distribution within the data set. ture convergence occurs when the weights and bias of the
All these networks use backpropagation for training. DNN settle into a state that is only optimal at a local level
Backpropagation uses gradient descent for error reduction, and misses out on the global minima of the entire multi-
by adjusting the weights based on the partial derivative of the dimensional space. Overfitting on the other hand describes a
error with respect to each weight. state when DNNs become highly tailored to a given training
Neural Network models can also be divided into the fol- data set at a fine grain level that it becomes unfit, rigid and
lowing two distinct categories: less adaptable for any other test data set.
1. Discriminative Along with different types of training, algorithms and
2. Generative architecture, we also have different machine learning frame-
Discriminative model is a bottom-up approach in which works (Table 1) and libraries that have made training models
data flows from input layer via the hidden layers to the output easier. These frameworks make complex mathematical func-
layer. They are used in supervised training for problems like tions, training algorithms and statistically modeling available
classification and regression. Generative models on the other without having to write them on your own. Some provide
hand are top-down and data flows in the opposite direction. distributed and parallel processing capabilities, and conve-
They are used in unsupervised pre-training and probabilistic nient development and deployment features. Figure 3 shows
distribution problems. If the input x and corresponding label y a graph with various deep learning libraries along with their
are given, a discriminative model learns the probability dis- Github stars from 2015-2018. Github is the largest hosting
tribution p(y|x), i.e., the probability of y given x directly, service provider of source code in the world [25]. Github
whereas a generative model learns the joint probability of stars are indicative of how popular a project is on Github.
p(x,y), from which P(y|x) can be predicted [20]. In general TensorFlow is the most popular DL library.
whenever labeled data is available discriminative approaches
III. DNN ARCHITECTURES
are undertaken as they provide effective training, and when
Deep neural network consists of several layers of nodes. Dif-
labeled data is not available generative approach can be
ferent architectures have been developed to solve problems in
taken [21].
different domains or use-cases. E.g., CNN is used most of the
Training can be broadly categorized into three types:
1. Supervised time in computer vision and image recognition, and RNN is
2. Unsupervised commonly used in time series problems/forecasting. On the
3. Semi-supervised other hand, there is no clear winner for general problems like
Supervised learning consists of labeled data which is used classification as the choice of architecture could depend on
to train the network, whereas unsupervised learning there multiple factors. Nonetheless [27] evaluated 179 classifiers
is no labeled data set, thus no learning based on feed- and concluded that parallel random forest or parRF_t, which
back. In unsupervised learning, neural networks are pre- is essentially parallel implementation of variation of decision
trained using generating models such as RBMs and later tree, performed the best. Below are three of the most common
could be fine-tuned using standard supervised learning algo- architectures of deep neural networks.
rithms. It is then used on test data set to determine pat- 1. Convolution Neural Network
terns or classifications. Big data has pushed the envelope 2. Autoencoder
even further for deep learning with its sheer volume and 3. Restricted Boltzmann Machine (RBM)
variety of data. Contrary to our intuitive inclination, there is 4. Long Short-Term Memory (LSTM)
no clear consensus on whether supervised learning is better
than the unsupervised learning. Both have their merits and A. CONVOLUTION NEURAL NETWORK
use cases. Reference [22] demonstrated enhance results with CNN is based on the human visual cortex and is the neural
unsupervised learning using unstructured video sequences network of choice for computer vision (image recognition)

VOLUME 7, 2019 53043

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 3. Github stars by Deep Learning Library [26].

TABLE 1. Popular deep learning frameworks and libraries. output layers. Fully connected layers that perform classifica-
tion follow the convolution layers. Sub-sampling or pooling
layers are often inserted between each convolution layers.
CNN’s takes a 2D n × n pixelated image as an input. Each
layer consists of groups of 2D neurons called filters or ker-
nels. Unlike other neural networks, neurons in each feature
extraction layers of CNN are not connected to all neurons in
the adjacent layers. Instead, they are only connected to the
spatially mapped fixed sized and partially overlapping neu-
rons in the previous layer’s input image or feature map. This
region in the input is called local receptive field. The lowered
number of connections reduces training time and chances of
overfitting. All neurons in a filter are connected to the same
number of neurons in the previous input layer (or feature map)
and are constrained to have the same sequence of weights and
biases. These factors speed up the learning and reduces the
memory requirements for the network. Thus, each neuron in a
specific filter looks for the same pattern but in different parts
of the input image. Sub-sampling layers reduce the size of
the network. In addition, along with local receptive fields and
shared weights (within the same filter), it effectively reduces
the network’s susceptibility of shifts, scale and distortions
of images [29]. Max/mean pooling or local averaging filters
are used often to achieve sub-sampling. The final layers of
CNN are responsible for the actual classifications, where
neurons between the layers are fully connected. Deep CNN
and video recognition. It is also used in other areas such can be implemented with multiple series of weight-sharing
as NLP, drug discovery, etc. As shown in Figure 4, a CNN convolution layers and sub-sampling layers. The deep nature
consists of a series of convolution and sub-sampling lay- of the CNN results in high quality representations while
ers followed by a fully connected layer and a normalizing maintaining locality, reduced parameters and invariance to
(e.g., softmax function) layer. Figure 4 illustrates the well- minor variations in the input image [30].
known 7 layered LeNet-5 CNN architecture devised by In most cases, backpropagation is used solely for training
LeCun et al. [28] for digit recognition. The series of mul- all parameters (weights and biases) in CNN. Here is a brief
tiple convolution layers perform progressively more refined description of the algorithm. The cost function with respect
feature extraction at every layer moving from input to to individual training example (x, y) in hidden layers can be

53044 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 4. 7-layer architecture of CNN for character recognition [28].

defined as [31]: Algorithm 1 CNN Backpropagation Algorithm Pseudo Code

1 1: Initialization weights to randomly generated value
J (W , b; x, y) =
||hw,b (x) − y| |2 (3) (small)
2
2: Set learning rate to a small value (positive)
The equation for error term δ for layer l is given by [31]: 3: Iteration n = 1; Begin
4: for n< max iteration OR Cost function criteria met, do
  0
δ (l) = (W (l) )T δ (l+1) .f (z(l) ) (4)
5: for image x1 to xi , do
where δ (l+1) is the error for (l + 1)th layer of a network 6: a. Forward propagate through convolution, pooling and
whose cost function is J (W , b; x, y). f (z(l) ) represents the
0 then fully conflected layers
derivate of the activation function. 7: b. Derive Cost Fuction value for the image
8: c.Calculate error term δ (l) with respect to weights for
∇w(l) J (W , b; x, y) = δ (l+1) (a(l+1) )T (5) each type of layers.
∇b(l) J (W , b; x, y) = δ (l+1) (6) 9: Note that the error gets propagated from layer to
layer in the following sequence
where a is the input, such that a(1) is the input for 1st layer 10: i.fully connected layer
(i.e., the actual input image) and a(l) is the input for l − th 11: ii.pooling layer
layer. 12: iii.convolution layer
Error for sub-sampling layer is calculated as [31]: 13: d.Calculate gradient ∇w(l) and ∇b(l) for weights ∇w(l)
  0 k k k
(l) (l)
δk = upsample (Wk )T δk
(l+1) (l)
· f (zk ) (7) and bias respectively for each layer
14: Gradient calculated in the following sequence
where k represent the filter number in the layer. In the sub- 15: i.convolution layer
sampling layer, the error has to be cascaded in the opposite 16: ii.pooling layer
direction, e.g., where mean pooling is used, upsample evenly 17: iii.fully connected layer
distributes the error to the previous input unit. And finally, 18: e.Update weights
(l) (l) (l)
here is the gradient w.r.t. feature maps [31]: 19: wji ← wji + 1wji
m 20: f.Update bias
(l) (l) (l)
21: bj ← bj + 1bj
 
(l) (l+1)
X
∇w(l) J (W , b; x, y) = (ai ) ∗ rot90 δk , 2 (8)
k
i−1
X (l+1)

∇b(l) J (W , b; x, y) = δk (9)
k a,b.
a,b
Reference [32] achieves this by leveraging Fourier trans-
(l) (l+1) formation to regularize inversion of the blurred images and
where (ai ) ∗ δk represents the convolution between error
and the i − th input in the l − th layer with respect to the denoising. Different implementations of CNN has shown
k − th filter. continuous improvement of accuracy in computer vision.
Algorithm 1 below represents a high-level description and The improvements are tested against the same benchmark
flow of the backpropagation algorithm as used in a CNN as (ImageNet) to ensure unbiased results.
it goes through multiple epochs until either the maximum Here are the well-known variation and implementation of
iterations are reached or the cost function target is met. the CNN architecture.
In addition to discriminative models such as image recog- 1. AlexNet:
nition, CNN can also be used for generative models such a. CNN developed to run on Nvidia parallel comput-
as deconvolving images to make blurry image sharper. ing platform to support GPUs

VOLUME 7, 2019 53045

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 5. Linear representation of a 2D data input using PCA.

2. Inception:
a. Deep CNN developed by Google
3. ResNet:
a. Very deep Residual network developed by
Microsoft. It won 1st place in the ILSVRC
2015 competition on ImageNet dataset.
4. VGG:
a. Very deep CNN developed for large scale image
recognition FIGURE 6. Training stages in autoencoder [36].

5. DCGAN:
a. Deep convolutional generative adversarial net- the least mean square error, which would then have the least
works proposed by [33]. It is used in unsupervised reconstruction error.
learning of hierarchy of feature representations in Autoencoders use encoder and decoder blocks of
input objects. non-linear hidden layers to generalize PCA to perform
dimensionality reduction and eventual reconstruction of the
B. AUTOENCODER original data. It uses greedy layer by layer unsupervised pre-
Autoencoder is a neural network that uses unsupervised algo- training and fin-tuning with backpropagation [35]. Despite
rithm and learns the representation in the input data set for using backpropagation, which is mostly used in supervised
dimensionality reduction and to recreate the original data set. training, autoencoders are considered unsupervised DNN
The learning algorithm is based on the implementation of the because they regenerate the input x (i) itself instead of a
backpropagation. different set of target values y(i) , i.e., y(i) = x (i) . Hinton
Autoencoders extend the idea of principal component et al. were able to achieve a near perfect reconstruction of
analysis (PCA). As shown in Figure 5, a PCA trans- 784-pixel images using autoencoder, proving that it is far
forms multi-dimensional data into a linear representation. better than PCA [36].
Figure 5 demonstrates how a 2D input data can be reduced to a While performing dimensionality reduction, autoencoders
linear vector using PCA. Autoencoders on the other hand can come up with interesting representations of the input vector in
go further and produce nonlinear representation. PCA deter- the hidden layer. This is often attributed to the smaller number
mines a set of linear variables in the directions with largest of nodes in the hidden layer or every second layer of the two-
variance. The p dimensional input data points are represented layer blocks. But even if there are higher number of nodes
as m orthogonal directions, such that m ≤ p and constitutes a in the hidden layer, a sparsity constraint can be enforced
lower (i.e., less than m) dimensional space. The original data on the hidden units to retain interesting lower dimension
points are projected into the principal directions thus omit- representations of the inputs. To achieve sparsity, some nodes
ting information in the corresponding orthogonal directions. are restricted from firing, i.e., the output is set to a value close
PCA focuses more on the variances rather than covariances to zero.
and correlations and it looks for the linear function with the Figure 6 shows single layer feature detector blocks
most variance [34]. The goal is to determine the direction with of RBMs used in pre-training, which is followed by

53046 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 8. Restricted Boltzmann machine.

build non-linear generative models from unlabeled data [39].

The goal is to train the network to increase a function
(e.g., product or log) of the probability of vector in the
visible units so it can probabilistically reconstruct the input.
It learns the probability distribution over its inputs. As shown
in Figure 8, RBM is made of two-layer network called the
visible layer and the hidden layer. Each unit in the visible
FIGURE 7. Autoencoder nodes.
layer is connected to all units in the hidden layer and there
are no connections between the units in the same layer.
unrolling [36]. Unrolling combines the stacks of RBMs to The energy (E) function of the configuration of the visible
create the encoder block and then reverses the encoder block and hidden units, (v, h) is expressed in the following way [40]:
to create the decoder section, and finally the network is fine- X X
E (v, h) = − ai vi − bj hj
tuned with backpropagation [36]. iεvisible jεhidden
X
Figure 7 illustrates a simplified representation of how − vi hj wij (12)
autoencoders can reduce the dimension of the input data i,j
and learn to recreate it in the output layer. Wang et al. [37] vi and hj are the vector states of the visible unit i and hidden
successfully implemented a deep autoencoder with stacks unit j. ai and bj represents the bias of visible and hidden units.
of RBM blocks similar to Figure 6 to achieve better mod- Wij denotes the weight between the respective visible and
eling accuracy and efficiency than the proper orthogonal hidden units.
decomposition (POD) method for dimensionality reduction The partition function, Z is represented by the sum of all
of distributed parameter systems (DPSs). The equation below possible pairs of visible and hidden vectors [40].
(2)
describes the average of activation function aj of jth unit of X
2nd layer when the xth input activates the neuron [38]. Z= e−E(v,h) (13)
v,h
1 Xm The probability of every pair of visible and hidden vectors
ρ̂ j = [aj (2) x (i) ](10) (10)
m i=1 is given by the following [40].
A sparsity parameter ρ is introduced such that ρ is very 1 −E(v,h)
close to zero, e.g., 0.03 and ρ̂ = ρ. To ensure that ρ̂ = ρ, p(v, h) =
Z
e (14)
a penalty term KL(ρ||ρ̂j ) is introduced such that the
Kullback–Leibler (KL) divergence term KL(ρ||ρ̂j ) = 0, The probability of a particular visible layer vector is pro-
if ρ̂j = ρ, else becomes large monotonically as the difference vided by the following [40].
between the two values diverges [38]. Here is the updated cost 1 X −E(v,h)
p (v) = e (15)
function [38]: Z h
s2
X As you can see from the equations above, the partition
J sparse (W , b) = J (W , b) + β KL(ρ||ρ̂ j )] (11) function becomes higher with lower energy function value.
j=1 Thus during the training process, the weights and biases of
where s2 equals the number of units in 2nd layer and β is the the network are adjusted to arrive at a lower energy and thus
parameter than controls sparsity penalty term’s weight. maximize the probability assigned to the training vector. It is
mathematically convenient to compute the derivative of the
C. RESTRICTED BOLTZMANN MACHINE (RBM)
log probability of a training vector.
Restricted Boltzmann Machine is an artificial neural net- ∂ log p(v)
= hvi hj idata − hvi hj imodel (16)
work where we can apply unsupervised learning algorithm to ∂wij
VOLUME 7, 2019 53047
 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

weights of recurrent output, bias and element-wise multipli-

cation respectively.
There is a smaller variation of the LSTM known as gated
recurrent units (GRU). GRUs are smaller in size than LSTM
as they don’t include the output gate, and can perform better
than LSTM on only some simpler datasets [44], [45].
LSTMs recurrent neural networks can keep track of long-
term dependencies. Therefore, they are great for learning
from sequence input data and building models that rely on
context and earlier states. The cell block of LSTM retains
pertinent information of previous states. The input, forget
and output gates dictates new data going into the cell, what
remains in the cell and the cell values used in the calculation
of the output of the LSTM block respectively [41], [43].
Naul et al. demonstrated LSTM and GRU based autoencoders
for automatic feature extractions [46].
FIGURE 9. LSTM block with memory cell and gates.
E. COMPARISON OF DNN NETWORKS
In the equation [40] above hvi hj idata and hvi hj imodel repre- Table 2 provides a compact summary and comparison of
sents the expectations under the respective distributions. the different DNN architectures. The examples of imple-
Thus, the adjustments in the weights can be denoted as mentations, applications, datasets and DL software frame-
follows [40], where  is the learning rate. works presented in the table are not implied to be exhaustive.
In addition, some of the categorization of the network archi-
1wij = (hvi hj idata − hvi hj imodel (17) tectures could be implemented in hybrid fashion. E.g., even
though RBMs are generative models and their training is
D. LONG SHORT-TERM MEMORY (LSTM)
considered unsupervised, they can have elements of discrim-
LSTM is an implementation of the Recurrent Neural Network inative model when training is finetuned with supervised
and was first proposed by Hochreiter et al. in 1997 [41]. learning. The table also provides examples of common appli-
Unlike the earlier described feed forward network architec- cations for using different architectures.
tures, LSTM can retain knowledge of earlier states and can
be trained for work that requires memory or state aware- IV. TRAINING ALGORITHMS
ness. LSTM partly addresses a major limitation of RNN, The learning algorithm constitutes the main part of Deep
i.e., the problem of vanishing gradients by letting gradients Learning. The number of layers differentiates the deep neural
to pass unaltered. As shown in the illustration in Figure 9, network from shallow ones. The higher the number of layers,
LSTM consists of blocks of memory cell state through which the deeper it becomes. Each layer can be specialized to detect
signal flows while being regulated by input, forget and output a specific aspect or feature.
gates. These gates control what is stored, read and written on As indicated by Najafabadi et al. [47], in case of image
the cell. LSTM is used by Google, Apple and Amazon in their (face) recognitions, first layer can detect edges and the second
voice recognition platforms [42]. can detect higher features such as various part of the face,
In figure 9, C, x, h represent cell, input and output values. e.g., ears, eyes, etc., and the third layer can go further up the
Subscript t denotes time step value, i.e., t −1 is from previous complexity order by even learning facial shapes of various
LSTM block (or from time t − 1) and t denotes current block persons. Even though each layer might learn or detect a
values. The symbol σ is the sigmoid function and tanh is defined feature, the sequence is not always designed for it,
the hyperbolic tangent function. Operator + is the element- especially in unsupervised learning. These feature extrac-
wise summation and x is the element-wise multiplication. tors in each layer had to be manually programmed prior
The computations of the gates are described in the equations to the development of training algorithms such as gradient
below [41], [43]. descent. These hand-crafted classifiers didn’t scale for lager
ft = σ (Wf xt + wf ht−1 + bf ) (18) dataset or adapt to variation in the dataset. This message was
echoed in the 1998 paper [28] by Yann Lecun et al., where
it = σ (Wi xt + wi ht−1 + bi ) (19) they demonstrate that systems with more automatic learning
ot = σ (Wo xt + wo ht−1 + bo ) (20) and reduced manually designed heuristics yields far better
ct = ft ⊗ ct−1 + it ⊗ σc (Wc xt + wc ht−1 + bc ) (21) pattern recognition.
ht = ot ⊗ σh (ct )(21) (22) Backpropagation provides representation learning method-
ology, where raw data can be fed without the need to manually
where f , i, o are the forget, input and output gate vectors massage it for classifiers, and it will automatically find the
respectively. W , w, b and ⊗ represent weights of input, representations needed for classification or recognition [6].

53048 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

TABLE 2. DNN network comparison table.

The goal of the learning algorithm is to find the optimal (or zero value) of a 2D function. To achieve this, we randomly
values for the weight vectors to solve a class of problem in a pick a point in the curve and slide to the right or left along
domain. the x-axis based on negative or positive value of the deriva-
Some of the well-known training algorithms are: tive or slope of the function at the chosen point until the value
1. Gradient Descent of the y-axis, i.e., function or f(x) becomes zero. The same
2. Stochastic Gradient Descent idea is used in gradient descent, where we traverse or descend
3. Momentum along a certain path in a multi-dimensional weight space if
4. Levenberg–Marquardt algorithm the cost function keeps decreasing and stop once the error rate
5. Backpropagation through time ceases to decrease. Newton’s method is prone to getting stuck
in local minima if the derivative of the function at the current
point is zero. Likewise, this risk is also present when using
A. GRADIENT DESCENT gradient descent on a non-convex function. In fact, the impact
Gradient descent (GD) is the underlying idea in most of is amplified in the multi-dimensional (each dimension repre-
machine learning and deep learning algorithms. It is based sents a weight variable) and multi-layer landscape of DNN
on the concept of Newton’s Algorithm for finding the roots and it result in a sub-optimal set of weights. Cost function

VOLUME 7, 2019 53049

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

C. MOMENTUM
In the standard SGD, learning rate is used as a fixed multiplier
of the gradient to compute step size or update to the weight.
This can cause the update to overshoot a potential minima,
if the gradient is too steep, or delay the convergence if the
gradient is noisy. Using the concept of momentum in physics,
the momentum algorithm presents a velocity v variable that
configured as an exponentially decreasing average of the
gradient [48]. This helps prevent costly descent in the wrong
direction. In the equation below, α ∈ [0, 1) is the momentum
parameter and  is the learning rate.
Velocity Update : v ← αv − g (24)
Actual Update : θ ← θ + v (25)

D. LEVENBERG-MARQUARDT ALGORITHM
Levenberg-Marquadt algorithm (LMA) is primarily used in
solving non-linear least squares problems such as curve fit-
ting. In least squares problems, we try to fit a given data
points with a function with the least amount of sum of the
squares of the errors between the actual data points and points
in the function. LMA uses a combination of gradient descent
and Gauss-Newton method. Gradient descent is employed
FIGURE 10. Error calculation in multilayer neural network [6]. to reduce the sum of the squared errors by updating the
parameters of the function in the direction of the steepest-
descent, while the Gauss-Newton method minimizes the error
is one half the square of the difference between the desired by assuming the function to be locally quadratic and finds the
output minus the current output as shown below. minimum of the quadratic [49].
If the fitting function is denoted by ŷ(t;p) and m data points
1
2
C= yexpected − yactual (23) denoted by (ti , yi ), then the squared error can be written
2 as [49]:
Backpropagation methodology uses gradient descent. Xm  y (ti ) − ŷ (ti ; p) 2
In backpropagation, chain rule and partial derivatives are x 2 (p) = (26)
employed to determine error delta for any change in the value i=1 σyi
= (y − ŷ (p))T W y − ŷ (p)


of each weight. The individual weights are then adjusted (27)
to reduce the cost function after every learning iteration of = yT Wy − 2yT W ŷ + ŷT W ŷ (28)
training data set, resulting in a final multi-dimensional (multi-
weight) landscape of weight values [6]. We process through where the measurement error for y (ti ), i.e., σyi is the inverse
all the samples in the training dataset before applying the of the weighting matrix Wii .
updates to the weights. This process is repeated until objective The gradient descent of the squared error function in
(aka cost function) doesn’t reduce any further. relation to the n parameters can be denoted as [49]:
Figure 10 shows the error derivatives in relation to outputs ∂ 2 ∂
x = 2(y − ŷ (p))T W y − ŷ (p)


in each hidden layer, which is the weighted summation of (29)
∂p ∂p
the error derivates in relation to the inputs in the unit in the
∂ ŷ (p)
 
above layer. E.g., when ∂E/∂zk calculated, the partial error = 2(y − ŷ (p))T W (30)
derivative with respect to wjk to is equal to yj ∂E/∂zk . ∂p
T
= 2(yŷ) W J (31)
hgd = αJT W y − ŷ


B. STOCHASTIC GRADIENT DESCENT (32)
Stochastic Gradient Descent (SGD) is the most common
variation and implementation of gradient descent. In gradient where J is the Jacobian matrix of size m × n used in place
descent, we process through all the samples in the training of the [∂ ŷ/∂p], and hgd is the update in the direction of the
dataset before applying the updates to the weights. While steepest gradient descent.
in SGD, updates are applied after running through a mini- The equation for the Gauss-Newton method update (hgn )
batch of n number of samples. Since we are updating the is as follows [49]:
weights more frequently in SGD than in GD, we can converge
h i
JT WJ hgn = JT W(y − ŷ) (33)
towards global minimum much faster.
53050 VOLUME 7, 2019
A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

The Levenberg- Marquardt update [hlm ] is generated by TABLE 3. Deep learning algorithm comparison table.
combining gradient descent and Gauss-Newton methods
resulting in the equation below [49]:
h i
JT WJ + λ diag(JT WJ) hlm = JT W(y − ŷ) (34)

E. BACKPROPAGATION THROUGH TIME

Backpropagation through time (BPTT) is the standard
method to train the recurrent neural network. As shown
in Figure 2b, the unrolling of RNN in time makes it appears
like a feedforward network. But unlike the feedforward net-
work, the unrolled RNN has the same exact set of weight val-
ues for each layer and represents the training process in time
domain. The backward pass through this time domain net-
work calculates the gradients with respect to specific weights
at each layer. It then averages the updates for the same weight
at different time increments (or layers) and changes them to
ensure the value of weights at each layer continues to stay
uniform.

F. COMPARISON OF DEEP LEARNING ALGORITHMS

Table 3 provides a summary and comparison of common deep
learning algorithms. The advantages and disadvantages are
presented along with techniques to address the disadvantages.
Gradient descent-based training is the most common type of
training. Backpropagation through time is the backpropaga-
tion tailored for recurrent neural network. Contrastive diver-
gence finds its use in probabilistic models such as RBMs.
Evolutionary algorithms can be applied to hyperparameter
optimizations or training models by optimizing weights.
Reinforcement learning could be used in game theory, multi-
agent systems and other problems where both exploitation
and exploration need to be optimized.

V. SHORTCOMINGS OF TRAINING ALGORITHMS

There are several shortcomings with the standard use of
B. LOCAL MINIMA
training algorithms on DNNs. The most common ones are
described here. Local minima is always the global minima in a convex
function, which makes gradient descent based optimization
A. VANISHING AND EXPLODING GRADIENTS
fool proof. Whereas in nonconvex functions, backpropaga-
tion based gradient descent is particularly vulnerable to the
Deep neural networks are prone to vanishing (or explod-
issue of premature convergence into the local minima. A local
ing) gradients due to the inherent way in which gradients
minima as shown in Figure 11, can easily be mistaken for
(or derivates) are computed layer by layer in a cascad-
global absolute minima.
ing manner with each layer contributing to exponentially
decreasing or increasing derivatives. Weights are increased
or decreased based on gradients to reduce the cost func- C. FLAT REGIONS
tion or error. Very small gradients can cause the network to Just like local minima, flat regions or saddle points
take a long time to train, whereas large gradients can cause (Figure 12) also pose similar challenge for gradient descent
the training to overshoot and diverge. This is made worse based optimization in nonconvex high-dimensional func-
by the non-linear activation functions like sigmoid and tanh tions. The training algorithm could potentially mislead by this
functions that squash the outputs to a small range. Since area as the gradient comes to a halt at this point.
change in weight have nominal effect on the output train-
ing could take much longer. This problem can be mitigated D. STEEP EDGES
using linear activation function like ReLu and proper weight Steep edges are another section of the optimization sur-
initialization. face area where the steep gradient could cause the gradient

VOLUME 7, 2019 53051

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 13. Overfitting in classification.

FIGURE 11. Gradient descent.

data set. One way to overcome overfitting is to choose the

number of neurons in the hidden layer wisely to match the
problem size and type. There are some algorithms that can be
used to approximate the appropriate number of neurons but
there is no magic bullet and the best bet is to experiment on
each use case to get an optimal value.

VI. OPTIMIZATION OF TRAINING ALGORITHMS

The goal of the DNN is to improve the accuracy of the
model on test data. Training algorithms aims to achieve the
end goal by reducing the cost function. The common root
cause of three out of five shortcomings mentioned above is
primarily due to the fact that the training algorithms assume
FIGURE 12. Flat (saddle point marked with black dot) region in a the problem area to be a convex function. The other problem
nonconvex function. is high number of nodes and the sheer possible combination
of weight values they can have. While weights are learned by
descent-based weight updates to overshoot and miss a poten- training on the dataset, there are additional crucial parameters
tial global minima. referred to as hyperparameters that aren’t directly learnt from
training dataset. These hyperparameters can take a range of
E. TRAINING TIME values and add complexity of finding the optimal architecture
Training time is an important factor to gauge the efficiency and model. There is significant room for improvement to the
of an algorithm. It is not uncommon for graduate students to standard training algorithms. Here are some of the popular
train their model for days or weeks in the computer lab. Most ways to enhance the accuracy of the DNNs.
models require exorbitant amount of time and large datasets
to train. Often times many of the samples from the datasets A. PARAMETER INITIALIZATION TECHNIQUES
do not add value to the training process and in some cases, Since the solution space is so huge, the initial parameters
they introduce noise and adversely affect the training. have an outsized influence on how fast or slow the train-
ing converges, if at all or if it prematurely converges to a
F. OVERFITTING
suboptimal point. Initialization strategies tend to be heuristic
As we add more neurons to DNN, it can undoubtedly model in nature. Reference [50] proposed normalized initialization
the network for more complex problems. DNN can lend itself where weights are initialized in the following manner.
to high conformability to training data. But there is also a high " √ √ #
risk of overfitting to the outliers and noise in the training data 6 6
as shown in Figure 13. This can result in delayed training and W ∼ U −√ ,√ (35)
nj +nj+1 nj +nj+1
testing times and result in the lower quality prediction on the
actual test data. E.g., in classification or cluster problems, Reference [51] proposed another technique called sparse
overfitting can create a high order polynomial output that initialization, where the number of non-zero incoming
separates the decision boundary for the training set, which weights were capped at a certain limit causing them to retain
will take longer and result in degraded results for most test high diversity and reduce chances of saturation.

53052 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

B. HYPERPARAMETER OPTIMIZATION
The learning rate and regularization parameters constitutes
the commonly used hyperparameters in DNN. Learning rate
determines the rate at which the weights are updated. The
purpose of regularization is the prevent overfitting and reg-
ularization parameter affects the degree of influence on the
loss function. CNN’s have additional hyperparameters i.e.,
number of filters, filter shapes, number of dropouts and
max pooling shapes at each convolution layer and number
of nodes in the fully connected layer. These parameters are
very important for training and modeling a DNN. Coming
up with an optimal set of parameter values is a challeng-
ing feat. Exhaustively iterating through each combination
of hyperparameter values is computationally very expensive.
For example, if training and evaluating a DNN with the full
dataset takes ten minutes, then with seven hyperparameters
each with eight potential values will take (87 × 10 min), i.e.,
20,971,520 minutes or almost 40 years to exhaustively train
and evaluate the network on all combinations of the hyperpa-
rameter values. Hyperparameter can be optimized with differ-
ent metaheuristics. Metaheuristics are nature inspired guiding
principles that can help in traversing the search space more
intelligently yet much faster than the exhaustive method.
Particle Swarm Optimization (PSO) is another type of
metaheuristic that can be used for hyperparameter optimiza-
tion. PSO is modeled around the how birds fly around in
search of food or during migration. The velocity and location
of birds (or particles) are adjusted to steer the swarm towards
better solution in the vast search space. Escalante et al. used
PSO for hyperparameter optimization to build a competitive
model that ranked among the top relative to other comparable
methods [52].
Genetic algorithm (GA) is a metaheuristic that is com-
monly used to solve combinatorial optimization problems.
It mimics the selection and crossover processes of species
reproduction and how that contributes to evolution and
improvement of the species prospect of survival. Figure 14a
shows a high-level diagram of the GA. Figure 14b illustrates
the crossover process where parts of the respective genetic FIGURE 14. (a) Genetic algorithm [53]. (b) Crossover in genetic algorithm.
sequence are merged from both the parents to form the new
genetic sequence in the children. The goal is to find a pop-
ulation member (a sequence of numbers resembling DNA population members. Hybridization is the process of mixing
nucleotides) that meets the fitness requirement. Each pop- the primary algorithm (GA in this case) with other operations,
ulation member represents a potential solution. Population like local search. Shrestha and Mahmood [53] incorporated
members are selected based on different methods, e.g., elite, 2-Opt local search method into GA to improve the search
roulette, rank and tournament. for optimal solution. Reference [55] postulates that correctly
Elite method ranks population members by fitness and performed exchanges (e.g., in GA) breeds innovation and
only uses high fitness members for the crossover process. results in creation solutions to hard problems just like in
The mutation process then makes random changes to the real life where collaboration and exchanges between indi-
number sequence and the entire process continues until a viduals, organizations and societies. In additional to GA,
desired fitness or maximum number of iterations are reached. other variations of evolution-based metaheuristics have also
References [53], [54] propose parallelization and hybridiza- been used to evolve and optimize deep learning architectures
tion of GA to achieve better and faster results. Parallelization and hyperparameters. E.g., [56] proposed CoDeepNEAT
provide both speedup and better results as we can periodically framework based on deep neuroevolution technique for
exchange population members between the distributed and finding an optimized architecture to match the task at
parallel operations of genetic algorithms on different set of hand.

VOLUME 7, 2019 53053

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

C. ADAPTIVE LEARNING RATES

Learning rates have a huge impact on training DNN. It can
speed up the training time, help navigate flat surfaces bet-
ter and overcome pitfalls of non-convex functions. Adaptive
learning rates allow us to change the learning rates for param-
eters in response to gradient and momentum. Several innova-
tive methods have been proposed. Reference [48] describes
the following:
1. Delta-bar Algorithm
2. AdaGrad
3. RMSProp
4. Adam
In Delta-bar algorithm, the learning rate of the param-
eter is increased if the partial derivative with respect to it
stays in the same sign and decreased if the sign changes.
AdaGrad is more sophisticated [57] and prescribes an
inversely proportional scaling of the learning rates to the
square root of the cumulative squared gradient. AdaGrad is
not effective for all DNN training. Since the change in the
FIGURE 15. Multilayer network training cost on MNIST dataset using
learning rate is a function of the historical gradient, AdaGrad different adaptive learning algorithms [58].
becomes susceptible to convergence.
RMSProp algorithm is a modification of AdaGrad algo-
rithm to make it effective in a nonconvex problem space.
RMSProd replaces the summation of squared gradient in
AdaGrad with exponentially decaying moving average of the
gradient, effectively dropping the impact of historical gradi-
ent [48]. Adam which denotes adaptive moment estimation
is the latest evolution of the adaptive learning algorithms that
integrates the ideas from AdaGrad, RMSProp and momen-
tum [58]. Just like AdaGrad and RMSProd, Adam provides
an individual learning rate for each parameter. Adam includes
the benefits of both the earlier methods does a better job
handling non-stationary objectives and both noisy and sparse
gradients problems [58]. Adam uses first moment (i.e., mean
as used in RMSProp) as well as second moments of the gra-
dients (uncentered variance) utilizing the exponential moving
average of squared gradient [58].
Figure 15 shows the relative performance of the various
adaptive learning rate mechanisms where Adam outperform
the rest. FIGURE 16. DNN with and without dropout.

D. BATCH NORMALIZATION models and later combining them to solve the larger model.
As the network is getting trained with variations to weights Greedy algorithms are commonly used in supervised pre-
and parameters, the distribution of actual data inputs at training of DNN.
each layer of DNN changes too, often making them all too
large or too small and thus making them difficult to train on F. DROPOUT
networks, especially with activation functions that implement There are few commonly used methods to lower the risk of
saturating nonlinearities, e.g., sigmoid and tanh functions. overfitting. In the dropout technique, we randomly choose
Iofee and Szegedy [59] proposed the idea of batch normal- units and nullify their weights and outputs so that they
ization in 2015. It has made a huge difference in improving do not influence the forward pass or the backpropagation.
the training time and accuracy of DNN. It updates the inputs Figure 16 shows a fully connected DNN on the left and a
to have a unit variance and zero mean at each mini-batch. DNN with dropout to the right. The other methods include
the use of regularization and simply enlarging the training
E. SUPERVISED PRETRAINING dataset using label preserving techniques. Dropout works
Supervised pretraining constitutes breaking down complex better than regularization to reduces the risk of overfitting
problems into smaller parts and then training the simpler and also speeds up the training process. Reference [60]

53054 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

proposed the dropout technique and demonstrated significant TABLE 4. DL algorithm shortcomings & resolution techniques.
improvement on supervised learning based DNN for com-
puter vision, computational biology, speech recognition and
document classification problems.

G. TRAINING SPEED UP WITH CLOUD AND GPU

PROCESSING
Training time is one of the key performance indicators of
machine learning. Cloud computing and GPUs lend them-
selves very well to speeding up the training process. Cloud
provides massive amounts of compute power and now all
major cloud vendors include GPU powered servers that can
easily be provisioned and used for training DNNs on demand
at competitive prices. Cloud vendor Amazon Web Services’
(AWS) P2 instances provides up to 40 thousand parallel GPU
cores and its P3 GPU instances are further optimized for
machine learning [61].

H. SUMMARY OF DL ALGORITHMS SHORTCOMINGS

AND RESOLUTIONS TECHNIQUES
Table 4 provides a summary of deep learning algorithm short-
comings and resolutions techniques. The table also lists the
cause and effect[s] of the shortcomings.

VII. ARCHITECTURES & ALGORITHMS –

IMPLEMENTATIONS
This section describes different implementations of neural
networks using a variety of training methods, network archi-
tectures and models. It also includes models and ideas that
have been incorporated into machine learning in general.

A. DEEP RESIDUAL LEARNING

The ability to add more layers to DNN has allowed us to solve
harder problems. Microsoft Research Asia (MSRA) applied a
100/1000 layer deep residual network (ResNet) on CIFAR-10
dataset and won 1st place in the ILSVRC 2015 competi-
tion with a 152-layer DNN on the ImageNet dataset [62].
Figure 17 demonstrates a simplified version of Microsoft’s
winning deep residual learning model. Despite the depth of
these networks, simply adding more layers to DNN does not
improve or guarantee results. To the contrary, it degrades
the quality of the solution. This makes training DNN not
so straight forward. The MSRA team was able to overcome
the degradation by making the hoping stacked layers match
a residual mapping instead of the desired mapping with the
following function [62]:
F (x) := H (x) − xv (36)
where F(x) is the residual mapping and H (x) is the desired FIGURE 17. Deep residual learning model by MSRA at Microsoft.

mapping, and then by recasting the desired mapping at the

end [62]. According to MSRA team, it is much easier to same and thus use each training examples the same number
optimize the residual mapping. of times. Simpson [63] argues that this assumption is invalid
and makes a case in his paper for the number of times a
B. ODDBALL STOCHASTIC GRADIENT DESCENT training examples is used to be proportional to its respective
All training data are not created equal. Some will have higher training error. So, if a training example has a higher error rate,
training error than the others. Yet, we assume that they are the it will be used to train the network higher number of times

VOLUME 7, 2019 53055

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

than the other training example. Simpson [63] proves his

methodology, termed Oddball Stochastic Gradient Descent
with a training set of 1000 video frames. Simpson [63] cre-
ated a training selection probability distribution for training
example based on the error value and pegged the frequency
of using the training example based on the distribution.

C. DEEP BELIEF NETWORK

Chen and Lin [23] highlights the fact that conventional neural
network can easily get stuck in local minima when the func-
tion is non-convex. They propose a DNN architecture called
large scale deep belief network (DBN) that uses both labeled
and unlabeled to learn feature representations. DBN are made
up of layers of RBM stacked together and learn probability
distribution of the input vectors. They employ unsupervised
pre-training and fine-tuned supervised algorithms and tech-
niques to mitigate the risk of getting trapped in local minima.
Below is the equation [23] for change in weights, where c is FIGURE 18. Learning multiple layers of representation.
the momentum factor and α is the learning rate, and v and h
are visible and hidden units respectively.

1wij (t + 1) = c1wij (t) + α(hvi hj idata − hvi hj imodel (37) generated today making information retrieval very chal-
lenging. Deep learning can help with semantic indexing to
Equation [23] for probability distribution for hidden and enable information to be more readily accessible in search
visible inputs. engines [14], [65]. This involves building models that provide
I
! relationships between documents and keywords the contain to
X
p(hj = 1|v; W) = σ wij vi + aj (38) make information retrieval more effective.

i=1 
XJ E. GENERATIVE TOP DOWN CONNECTION
p(vi = 1|h; W) = σ  wij hj + bi  (39) (GENERATIVE MODEL)
j=1 Much of the training is usually implemented with bottom-
up approach, where discriminatory or recognition models
D. BIG DATA are developed using backpropagation. A bottom-up model is
Big data provides tremendous opportunity and challenge for one that takes the vector representation of input objects and
deep learning. Big data is known for the 4 Vs (volume, veloc- computes higher level feature representations at subsequent
ity, veracity, variety). Unlike the shallow networks, the huge layer with a final discrimination or recognition pattern at the
volume and variety of data can be handled by DNNs and output layer. One of the shortcomings of backpropagation is
significantly improve the training process and the ability to that it requires labeled data to train. Geoffrey Hinton proposed
fit more complex models. On the flip side, the sheer veloc- a novel way of overcoming this limitation in 2007 [66].
ity of data that is generated in real-time can be daunting He proposed a multi-layer DNN that used generative top-
to process. Jajafabadi et al. [47] raises similar challenges down connection as opposed to bottom-up connection to
learning from real-time streaming data such as credit cards mimic the way we generate visual imagery in our dream
usage to monitor for fraud detection. They propose using without the actual sensory input. In top-down generative
parallel and distributed processing with thousands of CPU connection, the high-level data representation or the out-
cores. In addition, we should also use cloud providers that puts of the networks are used to generate the low-level raw
support auto-scaling based on usage and workload. Not all vector representations of the original inputs, one layer at a
data represent the same quality. In the case of computer time. The layers of feature representations learned with this
vision, images from constrained sources, e.g., studios are approach can then be further perfected either in generative
much easier to recognize that the ones from unconstrained models such as auto-encoders or even standard recognition
sources like surveillance cameras. Reference [64] proposes a models [66].
method to utilize multiple images of the unconstrained source In the generative model in Figure 18, since the correct
to enhance the recognition process. upstream cause of the events in each layer is known, a com-
Deep learning can help mine and extract useful patterns parison between the actual cause and the prediction made
from big data and build models for inference, prediction by the approximate inference procedure can be made, and
and business decision making. There is massive volumes the recognition weights, rij can be adjusted to increase the
of structured and unstructured data and media files getting probability of correct prediction.

53056 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 20. Pretraining of stacked & altered RBM to create a DBM [67].

FIGURE 19. Four-layer DBN & four-layer deep Boltzmann machine.

Here is the equation [66] for adjusting the recognition

weights rij .
!
X
1rij α hi hj − σ ( hi rij ) (40)
i

F. PRE-TRAINING WITH UNSUPERVISED DEEP

BOLTZMANN MACHINES
Vast majority of DNN training is based on supervised learn-
ing. In real life, our learning is based on both supervised and
unsupervised learning. In fact, most of our learning is unsu-
pervised. Unsupervised learning is more relevant in today’s
FIGURE 21. DBM getting initialized as deterministic neural network with
age of big data analytics because most raw data is unlabeled supervised fine-tuning [67].
and un-categorized [47]. One way to overcome the limitation
of backpropagation, where it gets stuck in local minima is to
incorporate both supervised and unsupervised training. It is pre-training) [67].
quite evident that top-down generative unsupervised learning  
is good for generalization because it is essentially adjusting X
the weights by trying to match or recreate the input data on p(vi = 1|h1 ) = σ  Wij1 hj  (41)
layer at a time [67]. After this effective unsupervised pre- j
training, we can always fine-tune it with some labeled data.  
Geoffrey Hinton and Ruslan Salakhutdinov describe multiple
X
p(h2m = 1|h1 ) = σ  2 1
Wjm hj (42)
layers of RBMs that are stacked together and trained layer by j
layer in a greedy, unsupervised way, essentially creating what !
is called the Deep Belief Network. They further modify stacks X X
to make them un-directed models with symmetric weights, p(h1j = 1|v, h ) = σ
2
Wij1 vi + 2 2
Wjm hm (43)
i m
thus creating the Deep Boltzmann Machines (DBM). Four
layered deep belief network and deep Boltzmann machines Post unsupervised pre-training, the DBM is converted
are shown in Figure 19. In [67] the DBM layers were pre- into a deterministic multi-layer neural network by fine-
trained one at a time using unsupervised method and then tuning the network with supervised learning using labeled
tweaked using supervised backpropagation on the MNIST data as demonstrated in Figure 21. The approximate
and NORB datasets as shown in Figure 20. They [67] received posterior distribution q(h|v) is generated for each input vec-
favorable results validating benefits of combining supervised tor and the marginals q(h2j = 1|v) are added as an addi-
and unsupervised learning methods. tional input for the network as shown in the figure above
Here are the equations showing probability distributions and subsequently, backpropagation is used to fine-tune the
over visible and two hidden units in DBM (after unsupervised network [67].

VOLUME 7, 2019 53057

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

G. EXTREME LEARNING MACHINE (ELM)

There have been other variations of learning methodologies.
While more layers allow us to extract more complex features
and patterns, some problems might be solved faster and bet-
ter with less number of layers. Reference [68] proposed a
four-layered CNN termed DeepBox that outperformed larger
networks in speed and accuracy. for evaluating objectness.
ELM is another type of neural network with just one hid-
den layer. Linear models are learnt from the dataset in a
single iteration by adjusting the weights between the hid-
den layer and the output, whereas the weights between the
input and the hidden layers are randomly initialized and
fixed [69].
ELM can obviously converge much faster than backprop-
agation, but it can only be applied to simpler problems of
classifications and regression. Since proposing ELM in 2006,
FIGURE 22. Pareto Frontier.
Buang-Bin Huang et al. came up with a multilayer version
of ELM in 2016 [70] to take on more complex problems.
They combined unsupervised multilayer encoding with the
random initialization of the weights and demonstrate faster
convergence or lower training time than the state of the art
multilayer perceptron training algorithm.

H. MULTIOBJECTIVE SPARSE FEATURE LEARNING MODEL

Gong et al. [71] developed a multi-objective sparse feature
learning (MO-SFL) model based on auto encoder, where they
used an evolutionary algorithm to optimize two competing
objectives of sparsity of hidden units and the reconstruction
error (input vendor of AE). It fairs better than models where
the sparsity is determined by human intervention or less than
optimal methods.
Since the time complexity of evolutionary algorithms are
high, they [71] utilize self-adaptive multi-objective differen-
tial evolution (DE) based on decomposition (Sa-MODE/D)
to cut down on time and demonstrate it has better results
FIGURE 23. Spectral clustering representation.
than standard AE (auto encoder), SR-RBM (Sparse response
RMB) and SESM (sparse encoding symmetric machine) by
testing with MNIST dataset and compare the results with I. MULTICLASS SEMI-SUPERVISED LEARNING BASED
other implementations. Their learning procedure continu- ON KERNEL SPECTRAL CLUSTERING
ously iterates between evolutionary optimization step and Mehrkanoon et al. [72] proposed a multiclass learning algo-
the stochastic gradient descent to optimize the reconstruction rithm based on Kernel Spectral Clustering (KSC) using both
error [71]. labeled and unlabeled data. The novelty of their proposal
• Step 1: Multi-objective optimization to select the most
is the introduction of regularization terms added to the cost
optimal point in the pareto frontier for both objectives function of KSC, which allow labels or membership to be
• Step 2: Optimize parameters θ and θ’ with stochastic
applied to unlabeled data examples. It is achieved in the
gradient descent in the following reconstruction error following way [72]:
function (of Auto Encoder), where D is the training data • Unsupervised learning based on kernel spectral cluster-

set and L (x,y) is the loss function with x representing the ing (KSC) is used as the core model
input and y representing the output, i.e., reconstructed • A regularization term is introduced and labels (from

input. labeled data) are added to the model

X Figure 23 illustrates data points in a spectral clustering
L(x, gθ 0 (f θ (x))) (44) representation. Spectral clustering (SC) is an algorithm that
x∈D divides the data points in a graph using Laplacian or double
Figure 22 shows a pareto frontier function that can be used derivative operation, whereas KSC is simply an extension
to achieve a compromise between two competing objectives of SC that uses Least Squares Support Vector Machines
functions. methodology [73].

53058 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

Since unlabeled data is more abundantly available relative during training but not during evaluation or testing, into the
to labeled data, it would be beneficial to make the most of it DNN training process.
with unsupervised or in this case semi-supervised learning.
L. GENETIC ALGORITHM
J. VERY DEEP CONVOLUTIONAL NETWORKS FOR
Genetic Algorithm is a metaheuristic that can be effectively
NATURAL LANGUAGE PROCESSING
used in training DNN. GA mimics the evolutionary processes
of selection, crossover and mutation. Each population mem-
Deep CNN have mostly been used in computer vision, where
ber represents a possible solution with a set of weights. Unlike
it is very effective. Conneau et al. [74] used it for the first
PSO, which includes only one operator for adjusting the solu-
time to NLP with up to 29 convolution layers. The goal is
tion, evolutionary algorithms like GA includes various steps,
to analyze and extract layers of hierarchical representations
i.e., selection, crossover and mutation methods [52]. Popu-
from words and sentences at the syntactic, semantic and con-
lation members undergo several iterations of selection and
textual level. One the major setbacks for lack of earlier deep
crossover based on known strategies to achieve better solution
CNN for NLP is because of deeper networks tend to cause
in the next iteration or generation. GA has undergone decades
saturation and degradation of accuracy. This is in addition to
of improvement and refinements since it was first proposed
the processing overhead of more layers. He et al. [62] states
in 1976 [78]. There are several ways to perform selec-
that the degradation is not caused by overfitting but because
tions, e.g., elite, roulette, rank, tournament [79]. There are
deeper systems are difficult to optimize. Reference [62]
about dozen ways to perform crossovers by Larrañaga et al.
addressed this issue with shortcut connections between the
alone [80]. Selection methodologies represent exploration of
convolution blocks to let the gradients to propagate more
the solution space and crossovers represent the exploitation of
freely and they, along with [74] were able to validate
the selected solution candidates. The goal is to get better solu-
the benefits of the shortcuts with 10/101/152-layers and
tion wider exploration and deeper exploitation. Additional
49 layers respectively. Conneau et al. [74] architecture con-
tweaking can be introduced with mutation. Parallel clusters
sists of series of convolution blocks separated by pooling
of GA can be executed independently in islands and few
that halved the resolution followed by k-max pooling and
members exchanged between the island every so often [81].
classification at the end.
In addition, we can also utilize local search such as greedy
algorithm, Nearest Neighbor or K-opt algorithm to further
K. METAHEURISTICS improve the quality of the solution.
Metaheuristics can be used to train neural networks to Lin et al. [82] demonstrated a successful incorporation
overcome the limitation of backpropagation-based learning. of GA that resulted in better classification accuracy and
When implementing metaheuristics as training algorithm, performance of a Polynomial Neural Network. Standard GA
each weight of the neural network connection is represented operations including selection, crossover and mutation were
by a dimension in the multi-dimensional solution search used on parameters that included partial descriptions (PDs)
space of the problem we are trying to solve. The goal is to of inputs in the first layer, bias and all input features [82].
come as near as possible to the optimal values of weights, GA was further enhanced with the incorporation of the
i.e., a location in the search space that represents the global concept of mitochondrial DNA (mtDNA). In evolution, it is
best solution. Particle Swarm Optimization (PSO) is a type quite evident from casual observation and simple reason that
of metaheuristic inspired by the movement of birds in the crossover of population members with too much similarity
sky consists of particles or candidate solutions move about does not yield much variance in the offspring. Likewise,
in a search space to reach a near optimal solution. In their we can infer that in GA, selection and crossover between
paper [75], N. Krpan and D. Jakobovic ran parallel imple- solutions that are very similar would not result is high degree
mentations using backpropagation and PSO. Their results of exploration of the multi-dimensional solution space.
demonstrate that while parallelization improves the efficacy In fact, it might run the risk of getting pigeonholed into a
of both algorithms, parallel backpropagation is efficient only restricted pattern.
on large networks, whereas parallel PSO has wider influence Diversity is the key to overcoming the risk of getting stuck
on various sizes of problems. in local minima. This risk can be mitigated by exploiting the
Similarly, Dong and Zhou [76] complemented PSO with idea of mtDNA. mtDNA represents one percent of the human
supervised learning control module to guide the search for chromosomes [83]. The concept of incorporating mitochon-
global minima of an optimization problem. The supervised drial DNA into GA was introduced by Shrestha and Mah-
learning module provided real-time feedback with back dif- mood [53]. They describe a way to restrict crossover between
fusion (BD) to retain diversity and social attractor renewal population members or solution candidates based proximity
to overcome stagnation [76]. Metaheuristics provide high on their mtDNA value [53]. Unlike the rest of the 99% DNA,
level guidance inspired by nature and applies them to solve mtDNA is only inherited from the female, thus it is a more
mathematical problems. In a similar way [77] proposes incor- continuous marker of lineage or genetic proximity. The
porating the concepts of intelligent teacher and privileged premise behind this is that offspring of population members
information, which is essentially extra information available with similar genetic makeup doesn’t help with overcoming

VOLUME 7, 2019 53059

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

M. NEURAL MACHINE TRANSLATION (NMT)

Neural Machine Translation is a turnkey solution used in
translation of sentences. While it provides some improvement
over the traditional Statistical machine translation (SMT),
it is not scalable for large models or datasets. It also requires
lot of computational power for training and translation, and
has difficult with rare words. For these reason, large tech
companies like Google and Microsoft have both improved on
NMT and have their own implementations of NMT, labeled
as Google Neural Machine Translation (GNMT) and Skype
Translator respectively. GMNT as shown in Figure 257 con-
sists of encoder and decoder LSTM blocks organized in layers
was presented in 2016 in [84]. It overcomes the shortcomings
of NMT with enhanced deep LSTM neural network that
includes 8 encoder and 8 decoder layers, and a method to
break down rare difficult words to infer their meaning. On
Conference on Machine Translation in 2014, GNMT received
results at par with state-of-the-art for English-to-French and
English-to-German language benchmarks [84].

N. MULTI-INSTANCE MULTI-LABEL LEARNING

Images in real life include multiple instances (objects)
and need multiple labels to describe them. E.g., a pic-
ture of an office space could include a laptop computer,
a desk, a cubicle and a person typing on the computer.
Zhou et al. [85] proposed MIML (Multi-Instance Multi-Label
learning) framework and corresponding MIMLBOOST and
MIMLSVM algorithms for efficient learning of individual
object labels in complex high level concepts, e.g., like the
office space. The goal is to learn f : 2x → 2y from dataset
{(X1 , Y1 ) , (X2 , Y2 ) , . . . , (Xm , Ym )}, where Xi ⊆ X represents
a set of instances {xi1 , xi2 , . . . xi,ni, }, xij ∈ X (j = 1, 2, . . . , ni ),
and Yi ⊆ Y represents a set of instances {yi1 , yi2 , . . . yi,li, },
yik ∈ Y (k = 1, 2, . . . , li ), where ni is the number of
instances in Xi and li is the number of labels in Yi [85].
MIMLBOOST uses category-wise decomposition into tra-
ditional single instance & single label supervised learning,
whereas MIMLSVN utilizes cluster-based feature transfor-
mation. So, instead of trying to learn the idea of complex
entities (e.g., office space), [85] took the alternate route and
learned the lower level individual objects and inferred the
higher level concepts.

O. ADVERSARIAL TRAINING
FIGURE 24. Continental model with mtDNA [53].
Machine learning training and deployment used to be done in
isolated computers, but now they are increasing being done in
the local minima. Figure 24 describes the parallel and dis- a highly interconnected commercial production environment.
tributed nature of their full implementation [53] along with Take a face recognition system where a network could be
the GA operators (selection, mutation and mtDNA incorpo- trained on a fleet of servers with a training dataset imported
rated crossover). The training process is enhanced [53] with from an external data source, and the trained model could
the implementation of continental model, where distributed be deployed on another server which accepts APIs calls with
servers run multiple threads, each running an instance of real time inputs (e.g., images of people entering a building)
GA with mtDNA. Population members are then exchanged and responds with matches. The interconnected architecture
between the servers after fixed number of iterations as shown exposes the machine learning to a wide attack surface. The
in Figure 24. real-time input or training dataset can be manipulated by an

53060 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 25. GNMT architecture [84] with encoder neural network on the left and decoder neural network on the right.

adversary to compromise the output (image match by the

network) or the entire model respectively.
Adversarial machine learning is a relatively new field of
research that takes into out these new threats to machine
learning. According to [86] adversaries (e.g., email spammer)
can exploit the lack of stationary data distribution and manip-
ulate the input (e.g., an actual spam email) as a normal email.
Reference [86] demonstrates these and other vulnerabilities
and discusses how application domain, features and data
distribution can be used to reduce the risk and impact of such
adversarial attacks.
FIGURE 26. GMM example with three components.
P. GAUSSIAN MIXTURE MODEL
Gaussian mixture model (GMM) is a statistical probabilistic where M is the number of number of gaussian components,
model used to represent multiple normal gaussian distribu- cm is the weight of the M -th gaussian, and (x; µm , 6m )
tions within a larger distribution using an EM (estimation represents the random variable x, which following the mean
maximization) algorithm in an unsupervised setting. E.g., vector µm .
a GMM could be used to represent the height distribution for
a large population group with two gaussian distributions, for Q. SIAMESE NETWORKS
male and female sub-groups. Figure 26 below demonstrates The purpose of siamese network is to determine the degree of
a GMM with three gaussian distributions within itself. similarity between two images. As shown in figure 27 below,
GMM has been used primarily in speech recognition and siamese network consists of two identical CNN networks
tracking objects in video sequences. GMM are very effec- with identical weights and parameters. The two images to be
tive in extracting speech features and modeling the prob- compared are passed separately through the two twin CNNs
ability density function to a desired level of accuracy as and the respective vector representations outputs are evalu-
long as we have sufficient components, and the estima- ated using contrastive divergence loss function. The function
tion maximization makes it easy to fit the model [87]. The is defined as following [88]:
probability density function for the GMM is given by the  −
→ − → 1
following [87]: L W , Y , X1 , X1 = (1 − Y ) (Dw )2
2
XM 1
p (x) = cm N (x; µm , 6m ) , (cm > 0) (45) + (Y ) (max(0, m − Dw ))2 (46)
m=1 2
VOLUME 7, 2019 53061
 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

FIGURE 28. Variational autoencoder.

including generating images in video games to de-noising

pictures.
In figure 28, x is the input and z is the encoded output
(latent variable). P(x) represents the distribution associated
with x. P(z) represents the distribution associated with z.
The goal is to infer P(z) based on P(z|x) that follows a
certain distribution. The mathematical derivation for VAEs
were originally proposed in [90]. Suppose we wanted to infer
P(z|x) based on some Q(z|x), then we can try to minimize the
KL divergence between the two:
XQ(z|x) log [ Q(z|x)
P(z|x) ]
DKL [Q (z|x) ||P (z|x)] = (47)
z
FIGURE 27. Siamese network.
Q (z|x)
= E [log [ ]] (48)
P (z|x)
Dw represents the Euclidean distance between the two = E [log Q (z|x) − logP (z|x)] (49)
output vectors as shown in figure 27. The output of the
where DKL is the Kullback–Leibler (KL) divergence and
contrastive divergence loss function, Y is either 1 (indicates
E represents expectation.
images are not the same) or 0 (indicates images are the
Using Baye’s rule:
same). m represents a margin value greater than 0. The idea
of siamese networks has been extended to come up with P (x|z) P(z)
P(z|x) = (50)
triplet networks, which includes three identical networks and P(x)
is used to assess the similarity of a given image with two other DKL [Q(z|x)||P(z|x)]
images. P(x|z)P(z)
= E [log Q(z|x) − Log ] (51)
Since the softmax layer outputs must match the number of P(x)
classes, a standard CNN becomes impractical for problems = E [log Q(z|x) − log P(x|z) − log P(z)] + log P(x)
that have large number of classes. This issue doesn’t apply (52)
to siamese network as the number of outputs of the softmax
in the twin networks doesn’t have the requirement to match To allow us to easily sample P(z) and generate new data,
the number of classes [89]. This ability to scale to many more we set P(z) to normal distribution, i.e., N (0, 1). If
PQ(z|x) is
classes for classification extends the use of siamese networks represented as gaussian with parameters µ(x) and (x), then
beyond what a traditional CNN is used for. Siamese network the KL divergence between Q(z|x) and P(z) can be derived in
can be used for handwritten check recognition, signature closed form as:
verification, text similarity, etc.
DKL [N (µ (x) , 6 (x)) ||N (0, 1)]
X
R. VARIATIONAL AUTOENCODERS = (1/2) (exp (6 (x)) + µ2 (x) − 1 − 6 (x)) (53)
k
As the name suggests, variational autoencoder (VAE), are
a type of autoencoder and consists of encoder and decoder S. DEEP REINFORCEMENT LEARNING
parts as shown in figure 28. It falls under the generative The primary idea about reinforcement learning is about mak-
model class of neural networks and are used in unsupervised ing an agent learn from the environment with the help of
learning. VAEs learn a low dimensional representation (latent random experimentation (exploration) and defined reward
variable) that model the original high dimensional dataset into (exploitation). It consists of finite number of states (si , rep-
a gaussian distribution. Kullback–Leibler (KL) divergence resenting agent and environment), actions (ai ) by the agent,
method is a good way to compare distributions. Therefore, probability (Pa ) of moving from one state to another based
the loss function in VAE is a combination of cross entropy on action ai , and reward Ra (si , si+1 ) associated with moving
(or mean squared error) to minimize reconstruction error and to the next state with action a. The goal is to balance and
KL divergence to make the compressed latent variable follow maximize the
current reward (R) and future reward (γ ·
a gaussian distribution. We then sample from the probability max[Q s0 , a0 ]) by predicting the best action as defined by
distribution to generate new dataset samples that are represen- this function Q (s, a) · γ in the equation represent a fixed
tative of the original dataset. It has found various applications discount factor. Q (s, a) is represented as the summation of

53062 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

current reward (R) and future reward (γ · max[Q s0 , a0 ]) as

of research and can lead to further advancement in machine
shown below. learning.
Q (s, a) = R + γ · max[Q s0 , a0 ]


(54)
VIII. CONCLUSION
Reinforcement learning is specifically suited for problems In this tutorial, we provided a thorough overview of the neural
that consists of both short-term and long-term rewards, e.g., networks and deep neural networks. We took a deeper dive
games like chess, go, etc. AlphaGo, Google’s program that into the well-known training algorithms and architectures.
beat the human Go champion also uses reinforcement learn- We highlighted their shortcomings, e.g., getting stuck in the
ing [91]. When we combine deep network architecture with local minima, overfitting and training time for large prob-
reinforcement learning, we get deep reinforcement learning lem sets. We examined several state-of-the-art ways to over-
(DRL), which can extend the use of reinforcement to even come these challenges with different optimization methods.
more complex games and areas such as robotics, smart grids, We investigated adaptive learning rates and hyperparameter
healthcare, finance etc. [92]. With DRL, problems that were optimization as effective methods to improve the accuracy
intractable with reinforcement learning can now be solved of the network. We surveyed and reviewed several recent
with higher number of hidden layers of deep networks and papers, studied them and presented their implementations and
reinforcement learning based Q-learning algorithm that max- improvements to the training process. We also included tables
imizes the reward for actions taken by the agent [13]. to summarize the content in a concise manner. The tables
provide a full view on how different aspects of deep learning
T. GENERATIVE ADVERSARIAL NETWORK (GAN) are correlated.
GANs consists of generative and discriminative neural net- Deep Learning is still in its nascent stage. There is
works. The generative network generates completely new tremendous opportunity for exploitation of current algo-
(fake) data based on input data (unsupervised learning) and rithms/architectures and further exploration of optimization
the discriminative network attempts to distinguish whether methods to solve more complex problems. Training is cur-
the data is real (from training set) or generated. The generative rently constrained by overfitting, training time and is highly
network is trained to increase the probability of deceiving susceptible to getting stuck in local minima. If we can
the discriminative network, i.e., to make the generated data continue to overcome these challenges, deep learning net-
indistinguishable from the original. GANs were proposed by works will accelerate breakthroughs across all applications
Goodfellow et al. [93] in 2014. It has been very popular as of machine learning and artificial intelligence.
it has many applications both good and bad. E.g., [94] were
able to successfully synthesize realistic images from text. CONFLICTS OF INTEREST
The authors declare no conflict of interest. The founding
U. MULTI-APPROACH METHOD FOR ENHANCING DEEP sponsors had no role in the design of the study; in the col-
LEARNING lection, analyses, or interpretation of data; in the writing of
Deep learning can be optimized at different areas. We dis- the manuscript, and/or in the decision to publish the results.
cussed training algorithm enhancements, parallel processing,
parameter optimizations and various architectures. All these ORCID
areas can be simultaneously implemented in a framework to Ajay Shrestha: http://orcid.org/0000-0001-5595-5953.
get the best results for specific problems. The training algo-
rithms can be finetuned at different levels by incorporating
REFERENCES
heuristics, e.g., for hyperparameter optimization. The time
[1] F. Rosenblatt, ‘‘The perceptron: A probabilistic model for information
to train a deep learning network model is a major factor to storage and organization in the brain,’’ Psychol. Rev., vol. 65, no. 6,
gauge the performance of an algorithm or network. Instead pp. 386–408, 1958.
of training the network with all the data set, we can pre- [2] M. Minsky and S. A. Papert, Perceptrons: An Introduction to Computa-
tional Geometry, Expanded Edition. Cambridge, MA, USA: MIT Press,
select a smaller but representative data set from the full 1969, p. 258.
training distribution set using instance selection methods [95] [3] G. Cybenko, ‘‘Approximation by superpositions of a sigmoidal function,’’
or Monte Carlo sampling [48]. An effective sampling method Math. Control, Signals Syst., vol. 2, no. 4, pp. 303–314, 1989.
can result in preventing overfitting, improving accuracy and [4] K. Hornik, ‘‘Approximation capabilities of multilayer feedforward net-
works,’’ Neural Netw., vol. 4, no. 2, pp. 251–257, 1991.
speeding up of the learning process without compromising on [5] P. J. Werbos, ‘‘‘Beyond Regression:’ New tools for prediction and analysis
the quality of the training dataset. Albelwi and Mahmood [96] in the behavioral sciences,’’ Ph.D. dissertation, Harvard Univ., Cambridge,
designed a framework that combined dataset reduction, MA, USA, 1975.
[6] Y. LeCun, Y. Bengio, and G. Hinton, ‘‘Deep learning,’’ Nature, vol. 521,
deconvolution network, correlation coefficient and an no. 7553, pp. 436–444, May 2015.
updated objective function. Nelder-Mead method was used [7] M. I. Jordan and T. M. Mitchell, ‘‘Machine learning: Trends, perspectives,
in optimizing the parameters of the objective function and and prospects,’’ Science, vol. 349, no. 6245, pp. 255–260, 2015.
the results were comparable to latest known results on the [8] A. Ng, ‘‘Machine learning yearning: Technical strategy for ai engineers in
the era of deep learning,’’ Tech. Rep., 2019
MNIST dataset [96]. Thus, combining optimizations at mul- [9] C. Metz, Turing Award Won by 3 Pioneers in Artificial Intelligence.
tiple levels and using multiple methods is a promising field New York, NY, USA: New York Times, 2019, p. B3.

VOLUME 7, 2019 53063

 A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

[10] K. Nagpal et al., ‘‘Development and validation of a deep learning algorithm [38] A. Ng. (Jul. 21, 2018). Autoencoders. UFLDL. [Online]. Available:
for improving Gleason scoring of prostate cancer,’’ CoRR, Nov. 2018. http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders
[11] S. Nevo, ‘‘ML for flood forecasting at scale,’’ CoRR, Jan. 2019. [39] Y. W. Teh and G. E. Hinton, ‘‘Rate-coded restricted Boltzmann machines
[12] A. Esteva et al., ‘‘Dermatologist-level classification of skin cancer with for face recognition,’’ in Proc. Adv. Neural Inf. Process. Syst., 2001,
deep neural networks,’’ Nature, vol. 542, no. 7639, p. 115, 2017. pp. 908–914.
[13] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, [40] G. E. Hinton, ‘‘A practical guide to training restricted Boltzmann
‘‘Deep reinforcement learning: A brief survey,’’ IEEE Signal Process. machines,’’ in Neural Networks: Tricks of the Trade, 2nd ed., G. Montavon,
Mag., vol. 34, no. 6, pp. 26–38, Nov. 2017. G. B. Orr, K.-R. Müller, Eds. Berlin, Germany: Springer, 2012,
[14] M. Gheisari, G. Wang, and M. Z. A. Bhuiyan, ‘‘A survey on deep learning pp. 599–619.
in big data,’’ in Proc. IEEE Int. Conf. Comput. Sci. Eng. (CSE), Jul. 2017, [41] S. Hochreiter and J. Schmidhuber, ‘‘Long Short-term Memory,’’ Neural
pp. 173–180. Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
[15] S. Pouyanfar, ‘‘A survey on deep learning: Algorithms, techniques, and [42] C. Metz, ‘‘Apple is bringing the AI revolution to your phone, in wired,’’
applications,’’ ACM Comput. Surv., vol. 51, no. 5, p. 92, 2018. Tech. Rep., 2016.
[16] R. Vargas, A. Mosavi, and R. Ruiz, ‘‘Deep learning: A review,’’ in Proc. [43] F. A. Gers, J. Schmidhuber, and F. Cummins, ‘‘Learning to forget:
Adv. Intell. Syst. Comput., 2017, pp. 1–11. Continual prediction with LSTM,’’ Neural Comput., vol. 12, no. 10,
[17] M. D. Buhmann, Radial Basis Functions. Cambridge, U.K.: Cambridge pp. 2451–2471, 2000.
Univ. Press, 2003, p. 270. [44] J. Chung. (2014). ‘‘Empirical evaluation of gated recurrent neural net-
[18] A. A. Akinduko, E. M. Mirkes, and A. N. Gorban, ‘‘SOM: Stochas- works on sequence modeling.’’ [Online]. Available: https://arxiv.org/abs/
tic initialization versus principal components,’’ Inf. Sci., vols. 364–365, 1412.3555
pp. 213–221, Oct. 2016. [45] K. Cho. (2014). ‘‘Learning phrase representations using RNN encoder-
[19] K. Chen, ‘‘Deep and modular neural networks,’’ in Springer Handbook decoder for statistical machine translation.’’ [Online]. Available: https://
of Computational Intelligence, J. Kacprzyk and W. Pedrycz, Eds. Berlin, arxiv.org/abs/1406.1078
Germany: Springer, 2015, pp. 473–494. [46] B. Naul, J. S. Bloom, F. Pérez, and S. van der Walt, ‘‘A recurrent neural
[20] A. Y. Ng and M. I. Jordan, ‘‘On discriminative vs. generative classifiers: network for classification of unevenly sampled variable stars,’’ Nature
A comparison of logistic regression and naive Bayes,’’ in Proc. 14th Int. Astron., vol. 2, no. 2, pp. 151–155, 2018.
Conf. Neural Inf. Process. Syst. Cambridge, MA, USA: MIT Press, 2001, [47] M. M. Najafabadi, F. Villanustre, T. M. Khoshgoftaar, N. Seliya, R. Wald,
pp. 841–848. and E. Muharemagic, ‘‘Deep learning applications and challenges in big
[21] C. M. Bishop and J. Lasserre, ‘‘Generative or discriminative? Getting the data analytics,’’ J. Big Data, vol. 2, no. 1, p. 1, Feb. 2015.
best of both worlds,’’ Bayesian Statist., vol. 8, pp. 3–24, Jan. 2007. [48] I. Goodfellow, Y. Bengio, and A. Courville, ‘‘Deep learning,’’ in Adaptive
[22] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, ‘‘Unsupervised learning Computation And Machine Learning. Cambridge, MA, USA: MIT Press,
of depth and ego-motion from video,’’ CoRR, Apr. 2017 2016, p. 775.
[23] X.-W. Chen and X. Lin, ‘‘Big data deep learning: Challenges and perspec- [49] H. P. Gavin, ‘‘The Levenberg-Marquardt method for nonlinear least
tives,’’ IEEE Access, vol. 2, pp. 514–525, 2014. squares curve-fitting problems,’’ Tech. Rep., 2016.
[24] Y. LeCun, K. Kavukcuoglu, and C. Farabet, ‘‘Convolutional networks [50] X. Glorot and Y. Bengio, ‘‘Understanding the difficulty of training deep
and applications in vision,’’ in Proc. IEEE Int. Symp. Circuits Syst., feedforward neural networks,’’ in Proc. 13th Int. Conf. Artif. Intell. Statist.,
May/Jun. 2010, pp. 253–256. 2010, pp. 249–256.
[25] G. Gousios, B. Vasilescu, A. Serebrenik, and A. Zaidman, ‘‘Lean GHTor-
[51] J. Martens, ‘‘Deep learning via Hessian-free optimization,’’ in Proc.
rent: GitHub data on demand,’’ in Proc. 11th Work. Conf. Mining Softw.
27th Int. Conf. Int. Conf. Mach. Learn. Haifa, Israel: Omnipress, 2010,
Repositories, Hyderabad, India, 2014, pp. 384–387.
pp. 735–742.
[26] AI-Index. (2019). Top Deep Learning Github Repositories. [Online].
[52] H. J. Escalante, M. Montes, and L. E. Sucar, ‘‘Particle swarm model
Available: https://github.com/mbadry1/Top-Deep-Learning
selection,’’ J. Mach. Learn. Res., vol. 10, pp. 405–440, Feb. 2009.
[27] M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, ‘‘Do we
[53] A. Shrestha and A. Mahmood, ‘‘Improving genetic algorithm with fine-
need hundreds of classifiers to solve real world classification problems?’’
tuned crossover and scaled architecture,’’ J. Math., vol. 2016, p. 10,
J. Mach. Learn. Res., vol. 15, no. 1, pp. 3133–3181, 2014.
Mar. 2016.
[28] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, ‘‘Gradient-based learn-
[54] K. Sastry, D. Goldberg, and G. Kendall, Genetic Algorithms. 2005.
ing applied to document recognition,’’ Proc. IEEE, vol. 86, no. 11,
pp. 2278–2324, Nov. 1998. [55] D. E. Goldberg, The Design of Innovation: Lessons from and for Competent
[29] Y. LeCun and Y. Bengio, ‘‘Convolutional networks for images, speech, Genetic Algorithms. Boston, MA, USA: Springer, 2013.
and time series,’’ in The Handbook of Brain Theory and Neural [56] R. Miikkulainen, ‘‘Evolving deep neural networks,’’ CoRR, Mar. 2017.
Networks, A. A. Michael, Ed. Cambridge, MA, USA: MIT Press, 1998, [57] J. Duchi, E. Hazan, and Y. Singer, ‘‘Adaptive subgradient methods for
pp. 255–258. online learning and stochastic optimization,’’ J. Mach. Learn. Res., vol. 12,
[30] G. W. Taylor, R. Fergus, Y. LeCun, and C. Bregler, ‘‘Convolutional learn- pp. 2121–2159, Jul. 2011.
ing of spatio-temporal features,’’ in Computer Vision. Berlin, Germany: [58] D. P. Kingma and J. Ba, ‘‘Adam: A method for stochastic optimization,’’
Springer, 2010. CoRR, Dec. 2014.
[31] A. Ng. (Jul. 21, 2018). Convolutional Neural Network. UFLDL. [59] S. Ioffe and C. Szegedy, ‘‘Batch normalization: Accelerating deep network
[Online]. Available: http://ufldl.stanford.edu/tutorial/supervised/ training by reducing internal covariate shift,’’ CoRR, Feb. 2015.
ConvolutionalNeuralNetwork/ [60] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
[32] C. J. Schuler, H. C. Burger, S. Harmeling, and B. Schölkopf, ‘‘A machine R. Salakhutdinov, ‘‘Dropout: A simple way to prevent neural networks
learning approach for non-blind image deconvolution,’’ in Proc. IEEE from overfitting,’’ J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958,
Conf. Comput. Vis. Pattern Recognit., Jun. 2013, pp. 1067–1074. 2014.
[33] A. Radford, L. Metz, and S. Chintala, ‘‘Unsupervised representation [61] AW Services. (Jul. 21, 2018). Amazon EC2 P2 & P3 Instances. Ama-
learning with deep convolutional generative adversarial networks,’’ CoRR, zon EC2 Instance Types. [Online]. Available: https://aws.amazon.com/ec2/
Nov. 2015. instance-types/p2/ and https://aws.amazon.com/ec2/instance-types/p3/
[34] I. T. Jolliffe, ‘‘Principal component analysis,’’ in Mathematics and Statis- [62] K. He, X. Zhang, S. Ren, and J. Sun, ‘‘Deep residual learning for image
tics, 2nd ed. New York, NY, USA: Springer, 2002, p. 487. recognition,’’ in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
[35] K. Noda, ‘‘Multimodal integration learning of object manipulation behav- Jun. 2016, pp. 770–778.
iors using deep neural networks,’’ in Proc. IEEE/RSJ Int. Conf. Intell. [63] A. J. R. Simpson, ‘‘Uniform learning in a deep neural network via ‘oddball’
Robots Syst., Nov. 2013, pp. 1728–1733. stochastic gradient descent,’’ CoRR, Oct. 2015.
[36] G. E. Hinton and R. R. Salakhutdinov, ‘‘Reducing the dimensionality of [64] L. Best-Rowden, H. Han, C. Otto, B. F. Klare, and A. K. Jain,
data with neural networks,’’ Science, vol. 313, no. 5786, pp. 504–507, ‘‘Unconstrained face recognition: Identifying a person of interest from
2006. a media collection,’’ IEEE Trans. Inf. Forensics Security, vol. 9, no. 12,
[37] M. Wang, H.-X. Li, X. Chen, and Y. Chen, ‘‘Deep learning-based model pp. 2144–2157, Dec. 2014.
reduction for distributed parameter systems,’’ IEEE Trans. Syst., Man, [65] T. A. Letsche and M. W. Berry, ‘‘Large-scale information retrieval with
Cybern., Syst., vol. 46, no. 12, pp. 1664–1674, Dec. 2016. latent semantic indexing,’’ Inf. Sci., vol. 100, nos. 1–4, pp. 105–137, 1997.

53064 VOLUME 7, 2019

A. Shrestha, A. Mahmood: Review of DL Algorithms and Architectures

[66] G. E. Hinton, ‘‘Learning multiple layers of representation,’’ Trends Cognit. [89] A. Shrestha and A. Mahmood, ‘‘Enhancing siamese networks training with
Sci., vol. 11, no. 10, pp. 428–434, Oct. 2007. importance sampling,’’ in Proc. 11th Int. Conf. Agents Artif. Intell. Prague,
[67] R. Salakhutdinov and G. Hinton, ‘‘Deep Boltzmann machines,’’ in Proc. Czech Republic: SciTePress, 2019, pp. 610–615.
12th Int. Conf. Artif. Intell. Statist., D. D. van and W. Max, Eds. 2009, [90] D. P. Kingma and M. Welling. (2013). ‘‘Auto-encoding variational Bayes.’’
pp. 448–455. [Online]. Available: https://arxiv.org/abs/1312.6114
[68] W. Kuo, B. Hariharan, and J. Malik, ‘‘DeepBox: Learning objectness with [91] D. Silver et al., ‘‘Mastering the game of go with deep neural networks and
convolutional networks,’’ CoRR, May 2015. tree search,’’ Nature, vol. 529, no. 7587, p. 484, 2016.
[69] G.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, ‘‘Extreme learning machine: The- [92] V. François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau,
ory and applications,’’ Neurocomputing, vol. 70, nos. 1–3, pp. 489–501, ‘‘An introduction to deep reinforcement learning,’’ CoRR, Dec. 2018.
2006. [93] I. J. Goodfellow et al. (2014). ‘‘Generative adversarial networks.’’
[70] J. Tang, C. Deng, and G.-B. Huang, ‘‘Extreme learning machine for [Online]. Available: https://arxiv.org/abs/1406.2661
multilayer perceptron,’’ IEEE Trans. Neural Netw. Learn. Syst., vol. 27, [94] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. (2016).
no. 4, pp. 809–821, Apr. 2015. ‘‘Generative adversarial text to image synthesis.’’ [Online]. Available:
[71] M. Gong, J. Liu, H. Li, Q. Cai, and L. Su, ‘‘A multiobjective sparse https://arxiv.org/abs/1605.05396
feature learning model for deep neural networks,’’ IEEE Trans. Neural [95] H. Brighton and C. Mellish, ‘‘Advances in instance selection for instance-
Netw. Learn. Syst., vol. 26, no. 12, pp. 3263–3277, Dec. 2015. based learning algorithms,’’ Data Mining Knowl. Discovery, vol. 6, no. 2,
[72] S. Mehrkanoon, C. Alzate, R. Mall, R. Langone, and J. A. K. Suykens, pp. 153–172, 2002.
‘‘Multiclass semisupervised learning based upon kernel spectral cluster- [96] S. Albelwi and A. Mahmood, ‘‘A framework for designing the architectures
ing,’’ IEEE Trans. Neural Netw. Learn. Syst., vol. 26, no. 4, pp. 720–733, of deep convolutional neural networks,’’ Entropy, vol. 19, no. 6, p. 242,
Apr. 2015. 2017.
[73] R. Langone, R. Mall, C. Alzate, and J. A. K. Suykens, ‘‘Kernel spectral
clustering and applications,’’ CoRR, May 2015.
[74] A. Conneau, H. Schwenk, L. Barrault, and Y. LeCun, ‘‘Very deep convo-
lutional networks for text classification,’’ CoRR, Jun. 2016.
[75] N. Krpan and D. Jakobovic, ‘‘Parallel neural network training with
OpenCL,’’ in Proc. 35th Int. Conv. MIPRO, May 2012, pp. 1053–1057.
AJAY SHRESTHA received the B.S. degree in
[76] W. Dong and M. Zhou, ‘‘A supervised learning and control method to computer engineering and the M.S. degree in com-
improve particle swarm optimization algorithms,’’ IEEE Trans. Syst., Man, puter science from the University of Bridgeport,
Cybern. Syst., vol. 47, no. 7, pp. 1135–1148, Jul. 2017. CT, USA, in 2002 and 2006, respectively, where he
[77] V. Vapnik and R. Izmailov, ‘‘Learning using privileged information: Simi- is currently pursuing the Ph.D. degree in computer
larity control and knowledge transfer,’’ J. Mach. Learn. Res., vol. 16, no. 1, science and engineering.
pp. 2023–2049, Jan. 2015. He has guest lectured at Pennsylvania State Uni-
[78] J. R. Sampson, Adaptation in Natural and Artificial Systems, vol. 18, no. 3, versity. He is also an Adjunct Faculty with the
J. H. Holland, Ed. Philadelphia, PA, USA: SIAM, 1976, pp. 529–530. School of Engineering, University of Bridgeport,
[79] N. M. Razali and J. Geraghty, ‘‘Genetic algorithm performance with and with Thermo Fisher Scientific, Branford,
different selection strategies in solving TSP,’’ in Proc. world Congr. Eng., CT, USA, as a Manager of Technical Operations. His research interests
2010, pp. 1–6. include machine learning and metaheuristics. He has served as a Technical
[80] P. Larrañaga, C. M. H. Kuijpers, R. H. Murga, I. Inza, and S. Dizdarevic, Committee Member of the International Conference on Systems, Computing
‘‘Genetic algorithms for the travelling salesman problem: A review of rep- Sciences and Software Engineering (SCSS). He received the Academic
resentations and operators,’’ Artif. Intell. Rev., vol. 13, no. 2, pp. 129–170,
Excellence Award and the Graduate Research Assistantship for his under-
Apr. 1999.
[81] D. Whitley, ‘‘A genetic algorithm tutorial,’’ Statist. Comput., vol. 4, no. 2,
graduate and graduate studies, respectively. He has been serving as the
pp. 65–85, Jun. 1994. Chapter Vice President and other officers of Upsilon Pi Epsilon (UPE),
[82] C.-T. Lin, M. Prasad, and A. Saxena, ‘‘An improved polynomial neural since 2014, and received the UPE Executive Council Award presented by
network classifier using real-coded genetic algorithm,’’ IEEE Trans. Syst., the UPE Executive Council, in 2016.
Man, Cybern., Syst., vol. 45, no. 11, pp. 1389–1401, Nov. 2015.
[83] Y. Guo et al., ‘‘The use of next generation sequencing technology to study
the effect of radiation therapy on mitochondrial DNA mutation,’’ Mutation
Res./Genetic Toxicol. Environ. Mutagenesis, vol. 744, no. 2, pp. 154–160,
2012. AUSIF MAHMOOD (SM’82) received the M.S.
[84] Y. Wu, ‘‘Google’s neural machine translation system: Bridging the gap and Ph.D. degrees in electrical and computer engi-
between human and machine translation,’’ CoRR, Sep. 2016.
neering from Washington State University, USA.
[85] Z.-H. Zhou, M.-L. Zhang, S.-J. Huang, and Y.-F. Li, ‘‘Multi-instance multi-
He is currently the Chair Person of the Com-
label learning,’’ Artif. Intell., vol. 176, no. 1, pp. 2291–2320, 2012.
[86] L. Huang, A. D. Joseph, B. Nelson, B. I. P. Rubinstein, and J. D. Tygar, puter Science and Engineering Department and a
‘‘Adversarial machine learning,’’ in Proc. 4th ACM Workshop Secur. Artif. Professor with the Computer Science and Engi-
Intell., Chicago, IL, USA, 2011, pp. 43–58. neering Department and the Electrical Engineering
[87] D. Yu and L. Deng, Automatic Speech Recognition: A Deep Learning Department, University of Bridgeport, Bridgeport,
Approach. London, U.K.: Springer, 2015. CT, USA. His research interests include parallel
[88] R. Hadsell, S. Chopra, and Y. LeCun, ‘‘Dimensionality reduction by learn- and distributed computing, computer vision, deep
ing an invariant mapping,’’ in Proc. IEEE Comput. Soc. Conf. Comput. Vis. learning, and computer architecture.
Pattern Recognit. (CVPR), Jun. 2006, pp. 1735–1742.

VOLUME 7, 2019 53065