Scribd
Upload
441 views

Autoencoders - Presentation

- An autoencoder is a type of neural network that learns to copy its input to its output. It has an internal hidden layer that encodes the input into a representation, and a decoder that reconstructs the output from this representation. - The goal of an autoencoder is to learn a representation (encoding) of the input data for dimensionality reduction or feature extraction, while minimizing the reconstruction error between the input and output. - Autoencoders can be undercomplete or overcomplete depending on whether the hidden layer has lower or higher dimensionality than the input. Regularization is needed for overcomplete autoencoders to avoid simply learning the identity function. - Autoencoders have applications in dimensionality reduction, noise removal, anomaly detection

Uploaded by

Phenil Buch
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
441 views

Autoencoders - Presentation

- An autoencoder is a type of neural network that learns to copy its input to its output. It has an internal hidden layer that encodes the input into a representation, and a decoder that reconstructs the output from this representation. - The goal of an autoencoder is to learn a representation (encoding) of the input data for dimensionality reduction or feature extraction, while minimizing the reconstruction error between the input and output. - Autoencoders can be undercomplete or overcomplete depending on whether the hidden layer has lower or higher dimensionality than the input. Regularization is needed for overcomplete autoencoders to avoid simply learning the identity function. - Autoencoders have applications in dimensionality reduction, noise removal, anomaly detection

Uploaded by

Phenil Buch
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 18

Autoencoders

Motivation – Representation Learning


• The success of machine learning heavily depends on data
representation (feature).
• Representation learning helps to improve performance of machine
learning models by extracting features from high dimensional data,
such as image or audio.
• Sometimes we would like to have a representation with a lower
dimension than the input dimension, to avoid complicated
calculations.
• But as we reduce dimensionality, there may be some loss of
information during compression.
Motivation – Representation Learning
• There are two broad parallel lines of representation learning approaches:
• Probabilistic graphical model – The hidden units are considered latent
random variables, for example: RBM.
• Neural networks – The hidden units in the model are considered as
computational graph, for example: Autoencoders.

• Manifold learning - Another approach for representation learning,


based on the associated geometry and the manifold assumption.
Autoencoders – An Introduction
• An Autoencoder is a type of Neural Network used to learn efficient data
encodings in an unsupervised manner.
• The goal of unsupervised feature learning(representation learning) is often
to discover low-dimensional features that captures some structure
underlying the high-dimensional input data.
• The aim of an Autoencoder is to learn a representation (encoding) for a set
of data, typically for dimensionality reduction. Along with the reduction
side, a reconstructing side is learnt, where the autoencoder tries to
generate from the reduced encoding a representation as close as possible
to its original input.
• Autoencoders are effectively used for solving many applied problems,
from face recognition to acquiring the semantic meaning of words.
• An autoencoder is a neural network that learns to
copy its input to its output. It has an internal
(hidden) layer that describes a code used to
represent the input, and it is constituted by two
main parts: an encoder that maps the input into
the code, and a decoder that maps the code to a
reconstruction of the original input.
• The Input and the output are approximately the
same.
• The middle hidden layer is the “Bottleneck Layer”
where the encoded representation is obtained.
• From the input layer to the bottleneck layer is the
“Encoder”
• From the bottleneck layer to the output layer is the
“Decoder”
Auto-encoder Basic Architecture

Target Target
= input
Y =X = input

Code Z Code

Input X Input

“Bottleneck” code “Overcomplete” code


i.e., low-dimensional, i.e., high-dimensional,
typically dense, always sparse,
distributed distributed
representation representation
Autoencoders
• Formally, we learn two functions:
• f(x) is the encoder
• g(h) is the decoder
• With respect to the reconstruction loss:
1
෍ 𝑥𝑘 − 𝑥ො𝑘 2
2
𝑘
• 𝑥 is the input and 𝑥ො is the output
(reconstructed input)
• h is the low dimensional
representation obtained at the
bottleneck layer
Undercomplete Autoencoder
• An Autoencoder is undercomplete if the dimension of the hidden layer is smaller than the
dimension of the input layer.
• The hidden layer “compresses” the input.

Hinton and Salakhutdinov,


“Reducing the Dimensionality of Data with Neural Networks”, 2006
Overcomplete Autoencoder
• An Autoencoder is overcomplete if the dimension of the hidden layer is larger than (or
equal to) the dimension of the input layer.
• In this case, we might have a duplication of the input in the features with perfect
reconstruction (identity function), without learning meaningful features.
• We use different forms of regularization to avoid this.
Applications of Autoencoders
• Dimensionality Reduction, Information Retrieval, Data Compression
• Noise Removal, Anomaly Detection, Face Recognition, Drug Discovery
• Training can use typical backpropagation methods

Trained
autoencoder
The Details
• In summary, basic autoencoder training consists in finding a value of
parameter vector minimizing reconstruction error:
• This minimization is usually carried out by stochastic gradient descent

12
Noise Removal from MNIST Dataset using Convolutional
Autoencoder
(CODE IN THE FOLLOWING SLIDE)

• By training an autoencoder, we are really training both the encoder


and the decoder at the same time.
• We can build an encoder and use it to compress MNIST digit images.
• Let’s also build a decoder so that we can decompress the compressed
image to the original image size.
• The UpSampling2D will repeats the rows and columns twice.
Effectively reversing the effect of the MaxPooling2D.
• In short, the UpSampling2D is doubling the height and width.
• We reshape the image data to the format the convolutional
autoencoder expects for training.
• We add random noises to the MINST image data and use them as
input for training.
• We train a new autoencoder with the noisy data as input and the
original data as expected output.
• During the training, the autoencoder learns to extract important
features from input images and ignores the image noises because the
labels have no noises.
Thank you

Prepared by: Phenil Buch


phenilbuch.ce@charusat.ac.in

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy