B.M.S.

COLLEGE OF ENGINEERING,
Bengaluru-560019.
Autonomous Institute, affiliated to VTU

A Project Report on

“Facial Sentiment Analysis Using Affective Computing”

Submitted in partial fulfilment of the requirements for the award of the degree of

MASTER OF TECHNOLOGY
in
DIGITAL COMMUNICATION ENGINEERING

by
MITHUN KUMAR K.P
1BM20LDC11

Under the guidance of

Dr. SIDDAPPAJI
Professor
Department of Electronics and Communication Engineering, BMSCE

2021 – 2022

DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING

B.M.S. COLLEGE OF ENGINEERING

Bengaluru-19
 BMS COLLEGE OF ENGINEERING
Autonomous Institute, affiliated to VTU
Bull Temple Road, Bengaluru – 560 019

DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING

CERTIFICATE

Certified that the project entitled Facial Sentiment Analysis Using AI Techniques is
a bonafide work carried out by MITHUN KUMAR K.P (1BM20LDC11) in partial
fulfillment for the award of Master of Technology in Digital Communication
Engineering of the Visvesvaraya Technological University, Belagavi during the
academic year 2021 -22. It is certified that, all corrections/suggestions indicated for
internal assessment have been incorporated in the report deposited in the departmental
library. The project report has been approved as it satisfies the academic requirements
in respect of project work prescribed for the said degree.

Guide Head of Department

Dr. SIDDAPPAJI Dr. SIDDAPPAJI

Professor and Head Professor and Head
Department of ECE Department of ECE
BMS College of Engineering BMS College of Engineering

Principal
Dr. S Muralidhara
BMS College of Engineering

External Viva
Name of the Examiners Signature with Date
1.
2.
 DECLARATION

I, MITHUN KUMAR K.P (1BM20LDC11), hereby declare that the dissertation work
entitled Facial Sentiment Analysis Using AI Techniques is a bonafide work and has been
independently carried out by me under the guidance of D r . S I D D A P P A J I , Professor and
Head Of Department, Department of Electronics and Communication Engineering, BMS
College of Engineering, Bengaluru, in partial fulfillment of the requirements of the degree of
Master of Technology in Digital Communication Engineering of Visvesvaraya
Technological University, Belagavi.

I further declare that, to the best of my knowledge and belief, this project has not been submitted
either in part or in full to any other university for the award of any degree.

Place: Bengaluru MITHUN KUMAR K.P

Date: 1BM20LDC11
 ACKNOWLEDGEMENTS

This project thesis has been done to brief the work titled “Facial Sentiment Analysis
Using AI Techniques” done as a part of M.Tech program.

I take this opportunity to express my profound gratitude and deep regards to Dr. S
Muralidhara, Principal, BMSCE for his cordial support.

I heartfully thank my guide Dr.SIDDAPPAJI, H.O.D Department of ECE, BMSCE for

his guidance and constant encouragement throughout the course of this project without which
this project would not be successful.

Last but not the least, I thank my family, friends and colleagues for their encouragement and help
in accomplishing the objective of the research work.
 ABSTRACT

With the advancements in machine and deep learning algorithms, the envision of various
critical real-life applications in computer vision becomes possible. One of the applications is
facial sentiment analysis. Deep learning has made facial expression recognition the most
trending research fields in computer vision area. Recently, deep learning-based facial
expression recognition (FER) models have suffered from various technological issues like
under-fitting or over-fitting. It is due to either insufficient training and expression data.
Human facial expressions are an integral means of displaying sentiments. Automatic analysis
of these unspoken sentiments has been an interesting and challenging task in the domain of
computer vision with its applications ranging across multiple domains including psychology,
product marketing, process automation etc. This task has been a difficult one as humans
differ greatly in the manner of expressing their sentiments through expressions. Previously
various machine learning techniques like Random forest, SVM etc. were used to predict the
sentiment using converted images. Deep learning has been instrumental in making
breakthrough progress in many fields of research including computer vision. This work
implements a convolutional neural network (CNN) based model for facial sentiment
detection.This system uses human face to identify the emotion once the emotion is identified
music is played related to human emotion.
Table of Contents
Abstract
List of figures i
List of abbreviations ii

1. INTRODUCTION.....................................................................................................................
2. LITERATURE REVIEW..........................................................................................................
3. PROBLEM STATEMENT.......................................................................................................
4. CONVOLUTION NEURAL NETWORKS (CNN)…………………………………….
5. METHODOLOGY....................................................................................................................
6. ARCHITECTURE AND LIBRARIES USED.....................................................................
7. DATASET............................................................................................................................
8. SIMULATION RESULTS........................................................................................................
9. CONCLUSION.........................................................................................................................
10. FUTURE SCOPE......................................................................................................................
REFERENCES.................................................................................................................................
APPENDIX…………………………………………………………………………….

Department of ECE,BMSCE i|Page

 List of figures

Figure Description of Figure Page

no. no.
1. FER Steps
2. The basic structure of a neuron
3. Architecture of CNN
4. Problem formulation of the project
5. Inception V3 model
6. TensorFlow
7. Keras
8. OpenCV
9. Image in Dataset FER-2013
10. Characteristic of happy expressions
11. Characteristic of anger expression
12. Characteristic of sad expression
13. Characteristic of fear expression
14. Expression of disgust
15. Characteristic expression of surprise
16. Characteristics of contempt expression
17. Training, Testing and Validation Data distribution

Department of ECE,BMSCE ii | P a g e
 List of tables

Table.No Description of Table Page. No

1. Performance of Inception V3

2. FER2013 dataset label breakdown

Department of ECE,BMSCE iii | P a g e

 List of abbreviations

Abbreviations Full Form

CNN Convolutional Neural Network

ML Machine Learning

NN Neural Network

FER 2013 Facial Emotion Recognition 2013 Dataset

FER Facial Emotion Recognition

AI Artificial Intelligence

DL Deep Learning

DCNN Deep Convolutional Neural Network

HCI Human-Computer Interaction

SVM Support Vector Machine

EV Expressional Vector

Department of ECE,BMSCE iv | P a g e
1.INTRODUCTION

Facial emotions are important factors in human communication that

help to understand the intentions of others. In general, people infer the
emotional state of other people, such as joy, sadness and anger, using facial
expressions and vocal tones. “2018 is the year when machines learn to grasp
human emotions” --Andrew Moore, the dean of computer science at carnegie
mellon.

With the advent of modern technology our desires went high and it binds no
bounds. In the present era a huge research work is going on in the field of digital
image and image processing. The way of progression has been exponential and it
is ever increasing. Image processing is a vast area of research in present day world
and its applications are very widespread. Image processing is the field of signal
processing where both the input and output signals are images. One of the most
important application of image processing is facial expression recognition. Our
emotion is revealed by the expressions in our face. Facial expressions plays an
important role in interpersonal communication. Facial expression is a non verbal
scientific gesture which gets expressed in our face as per our emotions. Automatic
recognition of facial expression plays an important role in artificial intelligence
and robotics and thus it is a need of the generation. Some application related to
this include personal identification and access control, videophone and
teleconferencing, forensic application, human-computer interaction, automated
surveillance, cosmetology and so on.
Emotions are efficacious and self-explanatory in normal day-to-day human
interactions. The most noticeable human emotion is through their facial expressions.
The facial expression recognition (FER) is quite complex and tedious but helps in
various applications areas such as healthcare, emotionally driven robots, and human-
computer interaction. Although the advancements in FER increases its effectiveness,
achieving a high accuracy is still a challenging task. The six most generic emotions of
a human are anger, happiness, sadness, disgust, fear, and surprise. In addition, the
emotion called contempt was added as one of the fundamental emotion.

Department of ECE,BMSCE v|Page

 Facial expression recognition (FER) is a baffling task and its accuracy is
completely dependent on the parameters selected, such as illumination factors,
occlusion, i.e., obstruction on the face like hand, age, and sun glasses. Researchers
of the field are taking these parameters into consideration while making facial
expression recognition (FER) models so that the considerable accuracy can be
achieved. The description of some important factors for facial expression
recognition (FER) is as follows.
• Illumination Factor: the light intensity falling on the object affects the
classification of the model. The textural values increase the false acceptance rate
due to either by minimizing the distance between classes or by increasing the
contrast.
• Expression Intensity: the expression recognition is highly dependent on the
intensity of the expression. The expression is recognized more accurately when
the expression is less subtle. It highly affects the accuracy of the model.
• Occlusion: if occlusion is present on an image then it becomes difficult for the
model to extract features from the occluded part due to inaccurate face alignment
and imprecise feature location. It also introduces noise to outliers and the
extracted features.

Facial emotion recognition (FER) is the technology that analyses facial

expressions from both static images and videos in order to reveal information on
one’s emotional state. The complexity of facial expressions, the potential use of
the technology in any context, and the involvement of new technologies such as
artificial intelligence raise significant privacy risks.

1.1 What is facial emotion recognition?

Facial emotion recognition is a technology used for analysing sentiments

by different sources, such as pictures and videos. It belongs to the family of
technologies often referred to as ‘Affective computing’, a multidisciplinary field of
research on computer’s capabilities to recognise and interpret human emotions and
affective states and it often builds on artificial intelligence technologies.

Facial emotion recognition (FER) analysis comprises three steps: a) face

Department of ECE,BMSCE vi | P a g e
 detection, b) facial expression detection, c) expression classification to an
emotional state. Emotion detection is based on the analysis of facial landmark
positions (e.g. end of nose, eyebrows). Furthermore, in videos, changes in those
positions are also analysed, in order to identify contractions in a group of facial
muscles. Depending on the algorithm, facial expressions can be classified to basic
emotions (e.g. anger, disgust, fear, joy, sadness, and surprise) or compound
emotions (e.g. happily sad, happily surprised, happily disgusted, sadly fearful,
sadly angry, sadly surprised). In other cases, facial expressions could be linked to
physiological or mental state of mind (e.g. tiredness or boredom). The source of
the images or videos serving as input to facial emotion recognition (FER)
algorithms vary from surveillance cameras to cameras placed close to advertising
screens in stores as well as on social media and streaming services or own personal
devices. Facial emotion recognition (FER) can also be combined with biometric
identification. Its accuracy can be improved with technology analysing different
types of sources such as voice, text, health data from sensors or blood flow
patterns inferred from the image. Potential uses of Facial emotion recognition
(FER) cover a wide range of applications, examples of which are listed here below
in groups by their application field.

Provision of personalised services

• Analyse emotions to display personalised messages in smart environments

• Provide personalised recommendations e.g. on music selection or cultural material

• Analyse facial expressions to predict individual reaction to movies

Customer behaviour analysis and advertising

• Analyse customers’ emotions while shopping focused on either goods or their

arrangement within the shop.
• Advertising signage at a railway station using a system of recognition and facial
tracking for marketing purposes

Healthcare

Department of ECE,BMSCE vii | P a g e

 • Detect autism or neurodegenerative diseases

• Predict psychotic disorders or depression to identify users in need of assistance

• Suicide prevention

• detect depression in elderly people

• observe patients conditions during treatment

Employment

• Help decision-making of recruiters

• Identify uninterested candidates in a job interview

• Monitor moods and attention of employees

Education

• Monitor students’ attention

• Detect emotional reaction of users to an educative program and adapt the learning
path

• Design affective tutoring system

• Detect engagement in online learning

Public safety

• Lie detectors

• Predictive screening of public spaces to identify emotions triggering potential

terrorism threat
• Analysing footage from crime scenes to indicate potential

motives in a crime.

Crime detection

Department of ECE,BMSCE viii |

Page
 • Detect and reduce fraudulent insurance claims

• Deploy fraud prevention strategies

• Spot shoplifters

Others

• Driver fatigue detection

• Detection of political attitudes.

1.2 FACIAL EXPRESSION RECOGNITION

Generally, face is a union of bones, facial muscles and skin tissues. When
these muscles contract, warped facial features are produced. Facial expressions are
the fastest means of communication while conveying any type of information. An
implementation of facial expression recognition may lead to a natural human-
machine interface. In 1978, Ekman and Frisen reported that facial expression acts
as a rapid signal that varies with contraction of facial features like eyebrows, lips,
eyes, cheeks etc., thereby affecting the recognition accuracy, also happy, sad, fear,
disgust, anger and surprise are six basic expressions which are readily recognized
across very different cultures. Facial expression recognition involves three steps
face detection, feature extraction and classification of expression.

Department of ECE,BMSCE ix | P a g e
 Figure 1. FER Steps [13]

1. FACE DETECTION

The pre-processing step for recognizing facial expressions is face detection. The
steps involved in converting a image to a normalized pure facial image for feature
extraction is detecting feature points, rotating to line up, locating and cropping the
face region using a rectangle, according to the face model. The face detection
involves methods for detecting faces in a single image.

2. FEATURE EXTRACTION

Feature extraction converts pixel data into a higherlevel representation of shape,

motion, color, texture, and spatial configuration of the face or its components.
Feature extraction generally reduces the dimensionality of the input space. The
reduction procedure should retain essential information as it is an important task
in pattern recognition system. Feature extraction can be done using various
techniques.

3.EXPRESSION CLASSIFICATION

Expression classification is performed by a classifier, which often consists of

models of pattern distribution, coupled to a decision procedure. Ekman defined
two main types of classes used in facial expression recognition those are action
units and prototypic facial expressions. There are various classification methods
which are used to extract expressions.

Department of ECE,BMSCE x|Page

In this work Deep Learning(DL) with Convolution neural network(CNN) is used. The
keras Application Programming Interface and OpenCV is used. OpenCV is used for
the automatic detection of faces and drawing boundary boxes around the face to
detect the correct expression.

This model is trained used 48x48 pixcl grey–scale images from Kaggle’s Facial
Expression Recognition 2013(FER-2013) dataset. This data consist of seven classes
of emotion which are happy, sad, anger, neutral, fear, disgust, and surprised. Once the
face is detected by the OpenCV then it matches with the dataset and then emotion of
the face is shown on top of the square box. After the emotion is detected music is
played related to the emotion.

Department of ECE,BMSCE xi | P a g e
 Facial Sentiment Analysis Using Affective
Computing

2. LITERATURE REVIEW

Akriti Jaiswal et.al,[1] Human Emotion detection from image is one of the most
powerful and challenging research task in social communication. Deep learning (DL) based
emotion detection gives performance better than traditional methods with image processing. It
presents the design of an artifificial intelligence (AI) system capable of emotion detection
through facial expressions. It discusses about the procedure of emotion detection, which includes
basically three main steps: face detection, features extraction, and emotion classifification. In this
Authors proposed a convolutional neural networks (CNN) based deep learning architecture for
emotion detection from images. The performance of the proposed method is evaluated using
dataset Facial emotion recognition challenge (FERC-2013).

Pranav E et.al,[2] The rapid growth of artificial intelligence has contributed a lot to the
technology world. As the traditional algorithms failed to meet the human needs in real time,
Machine learning and Deep learning algorithms have gained great success in different
applications such as classification systems, recommendation systems, pattern recognition etc.
Emotion plays a vital role in determining the thoughts, behaviour and feeling of a human. An
emotion recognition system can be built by utilizing the benefits of deep learning and different
applications such as feedback analysis, face unlocking can be implemented with good accuracy.
The main focus of this work is to create a Deep Convolutional Neural Network (DCNN) model
that classifies 5 different human facial emotions. The model is trained, tested and validated using
the manually collected image dataset.

Shuang Liu et.al,[3] Facial expressions play a significant role in nonverbal

communication. One of the important fields of study for Human-computer interaction (HCI) is
detecting facial expressions using emotion detection systems. Facial expressions are detected by
analyzing the discrepancy in various features of human faces such as color, posture, expression,
orientation etc. To detect the expression of a human face it is required to detect the different
facial features such as the movements of eye, nose, lips, etc. and then classify them by comparing
with trained data using a suitable classifier for expression recognition. The proposed facial
emotion recognition (FER) system uses viola jones algorithm for face detection, a pre-trained
convolutional neural network (CNN) called alexnet for feature extraction and the support vector
machine (SVM), a machine learning algorithm for classification of emotions. To demonstrate its

Department of ECE,BMSCE 12 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

application the proposed system is then applied for smart applications such as in smart cars to
control the speed of the automobiles based on driver’s emotions, regulating the drivers emotions
using mood lighting and displaying the detected emotions on an LCD display.

Balaji Balasubramanian et.al,[5] Human beings rely a lot on non-verbal communication

and facial emotion is a large part of it. Authors cover the datasets and algorithms that are used for
Facial Emotion Recognition (FER). The algorithms range from simple Support Vector Machines
(SVM) to complex Convolutional Neural Network (CNN). Authors explain these algorithms
through the fundamental research papers and go through their application to the task of FER.

Charvi Jain et.al,[6] The human face has peculiar and specific characteristics, therefore
it becomes difficult in understanding and identifying the facial expressions. It is easy to identify
the facial expression of particular person in any image sequence. If you look to automated
recognition system, however, the systems available are quite inadequate and incapable of
accurately identify emotions. The area of facial expression identification has many important
applications. It is an interactive tool between humans and computers. The user, without using the
hand can go-ahead with the facial expressions. Presently, the research on facial expression are on
the factors i.e. sad, happy, disgust, surprise, fear and angry. This research aims to detect faces
from any given image, extract facial features (eyes and lips) and classify them into 6 emotions
(happy, fear, anger, disgust, neutral, sadness).

Forugh Mozafari et.al,[8] The analysis of emotions and opinions is a new research
field that is aimed to identify opinions, feelings, assessments, tendencies and emotions of people
expressed in texts. The growing importance of emotion analysis has coincided with the growth of
social media, such as surveys, forums, blogs, twitter and social networks. Emotional analysis
systems are used in almost all businesses and social fields ,because opinions and emotions are
important for all human activities and have a serious impact on people’s emotions. Authors
discuss detecting and analyzing emotions based on similarity using VSM and STASIS methods.

Manoj Moolchandani et.al,[9] Facial emotion recognition (FER) gained considerable

interest among scientists working in areas like medical diagnosis, forensics and communication
technology. Numerous studies have already been conducted on facial emotion recognition (FER)
but none seems to deal with pose, illumination and age invariance simultaneously. This work
reviews several challenges for pose, illumination and age invariance for facial emotion
recognition (FER). Here Authors also provide a survey on deep learning methodologies used in

Department of ECE,BMSCE 13 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

facial emotion recognition (FER). It is noted that deep learning algorithms provide greater
accuracy than all other algorithms previously used.

Ma Xiaoxi et.al,[11] The interest on emotional computing has been increasing as many
applications were in demand by multiple markets. Authors mainly focuses on different learning
methods, and has implemented several methods: Support Vector Machine (SVM) and Deep
Boltzmann Machine (DBM) for facial emotion recognition. The training and testing data sets of
facial emotion prediction are from FERA 2015, and geometric features and appearance features
are combined together. Different prediction systems are developed and the prediction results are
compared. Authors aim is to design an suitable system for facial emotion recognition.

Fathimath Hafisa Hariz et.al,[12] Facial expressions play a significant role in

nonverbal communication. One of the important fields of study for Human-computer interaction
(HCI) is detecting facial expressions using emotion detection systems. Facial expressions are
detected by analyzing the discrepancy in various features of human faces such as color, posture,
expression, orientation etc. To detect the expression of a human face it is required to detect the
different facial features such as the movements of eye, nose, lips, etc. and then classify them by
comparing with trained data using a suitable classifier for expression recognition. The proposed
facial emotion recognition (FER) system uses viola jones algorithm for face detection, a pre-
trained convolutional neural network (CNN) called alexnet for feature extraction and the support
vector machine (SVM), a machine learning algorithm for classification of emotions. To
demonstrate its application this system is then applied for smart applications such as in smart cars
to control the speed of the automobiles based on driver’s emotions, regulating the drivers
emotions using mood lighting and displaying the detected emotions on an lcd display.

Ansamma John et.al,[15] Human emotion recognition plays a vital role in

interpersonal communication and human-machine interaction domain. Emotions are expressed
through speech, hand gestures and by the movements of other body parts and through facial
expression. Facial emotions are one of the most important factors in human communication that
help us to understand, what the other person is trying to communicate. People understand only
one-third of the message verbally, and two-third of it is through non-verbal means. There are
many face emotion recognition (FER) systems present right now, but in real-life scenarios, they
do not perform efficiently. Though there are many which claim to be a near-perfect system and to
achieve the results in favourable and optimal conditions. The wide variety of expressions shown

Department of ECE,BMSCE 14 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

by people and the diversity in facial features of different people will not aid in the process of
coming up with a system that is definite in nature. Hence developing a reliable system without
any flaws showed by the existing systems is a challenging task. This aims to build an enhanced
system that can analyse the exact facial expression of a user at that particular time and generate
the corresponding emotion.

Department of ECE,BMSCE 15 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

3. PROBLEM STATEMENT

Human emotions and intentions are expressed through facial expressions and deriving an
efficient and effective feature is the fundamental component of facial expression system.
Facial expressions convey non-verbal cues, which play an important role in interpersonal
communication. Automatic recognition of facial expressions can be an important component
of natural human-machine interfaces it may also be used in education, behavioral science and
in clinical practice. An automatic Facial Expression Recognition system needs to solve the
following problems: detection and location of faces in a cluttered scene, facial feature
extraction, and facial expression classification.

OBJECTIVE

To detect the emotion of a person using facial and visual clues. This system can be used to
identify the facial expression of the person. This application is highly useful in modern world
where humans rely on video calls more and more.

Department of ECE,BMSCE 16 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

4. Convolutional Neural Networks (CNN)

The fundamental building block of a neural network (NN) is a neuron. Forward

propagation of information through a neuron happens when inputs are multiplied by their
corresponding weights and then added together. This result is passed through a nonlinear
activation function along with a bias term which shifts the output. The output is between 0 and 1
which makes it suitable for problems with probabilities. The purpose of the activation function is
to introduce nonlinearities in the network since most real world data is nonlinear. The use of a
nonlinear function also allows neural networks (NN) to approximate complex functions.

Figure 2. The basic structure of a neuron [17]

Neurons can be combined to create a multi output NN. If every input has a connection to every
neuron it is called dense or fully connected. A deep NN has multiple hidden layers stacked on top
of each other and every neuron in each hidden layer is connected to a neuron in the previous
layer.

Department of ECE,BMSCE 17 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

4.1 CNN ARCHITECTURE

A typical architecture of a convolutional neural network contains an input layer,

some convolutional layers, some fully-connected layers, and an output layer. CNN is
designed with some modification on LeNet Architecture. It has 6 layers without considering
input and output. The architecture of the Convolution Neural Network (CNN) used in this
project is shown in the following figure.

Figure 3. Architecture of CNN [5]

1.Input Layer:
The input layer has pre-determined, fixed dimensions, so the image must be pre-processed
before it can be fed into the layer. Normalized gray scale images of size 48 X 48 pixels from
Kaggle dataset are used for training, validation and testing. For testing propose laptop
webcam images are also used, in which face is detected and cropped using OpenCV Haar
Cascade Classifier and normalized.

2.Convolution and Pooling (ConvPool) Layers:

Convolution and pooling is done based on batch processing. Each batch has N images and CNN
filter weights are updated on those batches. Each convolution layer takes image batch input of
four dimension N x Color-Channel x width x height. Feature map or filter for convolution are
also four dimensional (Number of feature maps in, number of feature maps out, filter width, filter

Department of ECE,BMSCE 18 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

height). In each convolution layer, four dimensional convolution is calculated between image
batch and feature maps. There are two types of pooling, namely max pooling and average
pooling. Max pooling returns the maximum value from the portion of the image covered by the

kernel, while average pooling returns the average of the corresponding values.

Pool size of (2x2) is 12 taken, which splits the image into grid of blocks each of size 2x2 and
takes maximum of 4 pixels. After pooling only height and width are affected. Two convolution
layer and pooling layer are used in the architecture. At first convolution layer size of input
image batch is Nx1x48x48. Here, size of image batch is N, number of color channel is 1 and
both image height and width are 48 pixel. Convolution with feature map of 1x20x5x5 results
image batch is of size Nx20x44x44. After convolution pooling is done with pool size of 2x2,
which results image batch of size Nx20x22x22. This is followed by second convolution layer
with feature map of 20x20x5x5, which results image batch of size Nx20x18x18. This is
followed by pooling layer with pool size 2x2, which results image batch of size Nx20x9x9.

3.Fully Connected Layer

This layer is inspired by the way neurons transmit signals through the brain. It takes a large
number of input features and transform features through layers connected with trainable
weights. Two hidden layers of size 500 and 300 unit are used in fully-connected layer. The
weights of these layers are trained by forward propagation of training data then backward
propagation of its errors. Back propagation starts from evaluating the difference between
prediction and true value, and back calculates the weight adjustment needed to every layer
before. Training speed and the complexity of the architecture cab be controlled by tuning the
hyper-parameters, such as learning rate and network density. Hyper-parameters for this layer
include learning rate, momentum, regularization parameter, and decay.

The output from the second pooling layer is of size Nx20x9x9 and input of first hidden layer
of fully-connected layer is of size Nx500. So, output of pooling layer is flattened to Nx1620
size and fed to first hidden layer. Output from first hidden layer is fed to second hidden layer.
Second hidden layer is of size Nx300 and its output is fed to output layer of size equal to
number of facial expression classes.

Department of ECE,BMSCE 19 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

4.Output Layer
Output from the second hidden layer is connected to output layer having seven distinct
classes. Using Softmax activation function, output is obtained using the probabilities for each
of the seven class. The class with the highest probability is the predicted class.

5.METHODOLOGY

The facial expression recognition system is implemented using convolutional neural network
with the goal to improve the process of facial sentiment analysis systems. FER systems can
be either static or dynamic based on image. Static FER considers only the face point location
information from the feature representation of a single image, whereas, the dynamic image
FER considers the temporal information with continuous frames. The static FER process over
is exhibited with description of steps as follows.

• Dataset: To avoid over-fitting, the following FER algorithms are discussed, which needs
extensive training data. A dataset must have well-defined emotion tags of facial expression is
essential for testing, training, and validating the algorithms for the development of FER.
These datasets contain a sequence of images with distinct emotions.
• Pre-processing: this step pre-processes the dataset by removing noise and data compression.
Various steps involved in data pre-processing are:
(i) Facial detection is the power to detect the location of the face in any image or frame.
It is often considered as a special case of object-class detection, which determines
whether the face is present in an image or not,
(ii) Dimension reduction is used to reduce the variables by a set of principal variables. If
the number of features is more, then it gets tougher to visualize the training set
and to work on it.
(iii) Normalization: It is also known as feature scaling. After the dimension reduction
step, reduced features are normalized without distorting the differences in the range of
values of features. There are various normalization methods, namely z normalization,

Department of ECE,BMSCE 20 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

min-max normalization, unit vector normalization, which improves the numerical

stability and speeds up the training of the model.
• Feature extraction: It is the process of extracting features that are important for FER. It
results in smaller and richer sets of attributes that contain features like face edges, corners,
diagonal, and other important information such as distance between lips and eyes, the
distance between two eyes, which helps in speedy learning of trained data.
• Emotion classification: it involves the algorithms to classify the emotions based on the
extracted features. The classification has various methods, which classifies the images into
various classes. The classification of a FER image is carried out after passing through pre-
processing steps of face detection and feature extraction.

Figure 4. Problem formulation of the project

Convolutional neural network (CNN) is the most popular way of analyzing images. CNN is
different from a multi-layer perceptron (MLP) as they have hidden layers, called convolutional
layers. The proposed method is based on a two-level CNN framework. The first level
recommended is background removal, used to extract emotions from an image. Here, the
conventional CNN network module is used to extract primary expressional vector (EV). The
expressional vector (EV) is generated by tracking down relevant facial points of importance.

Department of ECE,BMSCE 21 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

EV is directly related to changes in expression. The EV is obtained using a basic perceptron

unit applied on a background-removed face image. Each of the convolutional layers receives
the input data (or image), transforms it, and then outputs it to the next level. This
transformation is convolution operation.

6.ARCHITECTURE AND LIBRARIES USED

6.1 Inception V3

The Inception V3 is a deep learning model based on Convolutional Neural Networks, which is
used for image classification. The inception V3 is a superior version of the basic model
Inception V1 which was introduced as GoogLeNet in 2014. As the name suggests it was
developed by a team at Google.

What makes the Inception V3 model better?

The inception V3 is just the advanced and optimized version of the inception V1 model. The
Inception V3 model used several techniques for optimizing the network for better model
adaptation.
1. It has higher efficiency
2. It has a deeper network compared to the Inception V1 and V2 models, but its speed isn't
compromised.
3. It is computationally less expensive.
4. It uses auxiliary Classifiers as regularizes.

Department of ECE,BMSCE 22 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

The inception v3 model was released in the year 2015, it has a total of 42 layers and a lower
error rate than its predecessors. Let's look at what are the different optimizations that make
the inception V3 model better.

The major modifications done on the Inception V3 model are:

1. Factorization into Smaller Convolutions

2.Spatial Factorization into Asymmetric Convolutions
3.Utility of Auxiliary Classifiers
4.Efficient Grid Size Reduction

Fig 5. Inception V3 model [19]

In total, the inception V3 model is made up of 42 layers which is a bit higher than the previous
inception V1 and V2 models. But the efficiency of this model is really impressive.

Performance of Inception V3

As expected the inception V3 had better accuracy and less computational cost compared to
the previous Inception version.

Department of ECE,BMSCE 23 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Table 1. Performance of Inception V3 [19]

6.2 Libraries used

Tensorflow :

TensorFlow is Google’s gift to the developers involved in Machine Learning. It makes the
power of Deep Learning accessible to the people in pursuit. Google has a beginner as well as
an advanced tutorial which introduce you to both ML and TF concurrently to solve a multi-
feature problem --- character recognition.

Figure 6. TensorFlow

Department of ECE,BMSCE 24 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

TensorFlow is an open-sourced library that’s available on GitHub. It is one of the more famous
libraries when it comes to dealing with Deep Neural Networks. The primary reason behind the
popularity of TensorFlow is the sheer ease of building and deploying applications using
TensorFlow. The sample projects provided in the GitHub repository are not only powerful but
also written in a beginner-friendly way.

Keras:

Keras is a high-level library that’s built on top of Theano or TensorFlow. It provides a scikit-
learn type API (written in Python) for building Neural Networks. Developers can use Keras
to quickly build neural networks without worrying about the mathematical aspects of tensor
algebra, numerical techniques, and optimization methods.
The key idea behind the development of Keras is to facilitate experimentations by fast
prototyping. The ability to go from an idea to result with the least possible delay is key to
good research.
This offers a huge advantage for scientists and beginner developers alike because they can
dive right into Deep Learning without getting their hands dirty with low-level computations.
The rise in the demand for Deep Learning has resulted in the rise in demand for people
skilled in Deep Learning.

Figure 7. Keras
Every organization is trying to incorporate Deep Learning in one way or another, and Keras
offers a very easy to use as well as intuitive enough to understand API which essentially helps
you test and build Deep Learning applications with least considerable efforts. This is good
because Deep Learning research is such a hot topic right now and scientists need a tool to try
out their ideas without wasting time on putting together a Neural Network model.

Department of ECE,BMSCE 25 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Salient Features of Keras

 Keras is a high-level interface and uses Theano or Tensorflow for its
backend.
 It runs smoothly on both CPU and GPU.
 Keras supports almost all the models of a neural network – fully connected,
convolutional, pooling, recurrent, embedding, etc. Furthermore, these
models can be combined to build more complex models.
 Keras, being modular in nature,  is incredibly expressive, flexible for
innovative research.
 Keras is a completely Python-based framework, which makes it easy to
debug and explore.

OpenCV
OpenCV is a Python library which is designed to solve computer vision problems. OpenCV
was originally developed in 1999 by Intel but later it was supported by Willow Garage.

OpenCV supports a wide variety of programming languages such as C++, Python, Java etc.
Support for multiple platforms including Windows, Linux, and MacOS.

OpenCV Python is nothing but a wrapper class for the original C++ library to be used with
Python. Using this, all of the OpenCV array structures gets converted to/from NumPy arrays.

This makes it easier to integrate it with other libraries which use NumPy. For example,
libraries such as SciPy and Matplotlib.

Department of ECE,BMSCE 26 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Figure 8. OpenCV

Face Detection Using OpenCV

Step 1: Considering the prerequisites, an image is required, to begin with. Later cascade
classifier is created which will eventually give us the features of the face.
Step 2: This step involves making use of OpenCV which will read the image and the features
file. So at this point, there are NumPy arrays at the primary data points.
Step 3: This final step involves displaying the image with the rectangular face box.

7.DATASET
7.1 Facial Expression Recognition 2013 (FER-2013)
The 2013 Facial Expression Recognition dataset (FER- 2013) is a dataset provided by
Kaggle, introduced at the International Conference on Machine Learning (ICML) in 2013
introduced by Pierre-Luc Carrier and Aaron Courvill. In this dataset, each face has been
categorized based on emotion categories, where the FER-2013 dataset is a grayscale image
measuring 48*48 pixel for each image. The total FER-2013 dataset is 35,887 consisting of 7
(seven) different types of micro expression.

Department of ECE,BMSCE 27 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Figure 9. Image in Dataset FER-2013 [18]

A micro expression is a facial expression that can easily observe and distinguish it as a
communication method in social psychology. Facial expressions provide information about
emotions, convey our intentions and goals, and play an essential role in human interaction. The
ability to recognize and understand facial expressions automatically facilitates the intended
communication.

The process in the classifification of human facial expressions consists of three stages:face
detection, feature extraction, and facial expression classifification. In this study, the authors
applied a system that could classify facial expressions at a macro level, consisting of 7 (seven)
basic human expressions:

1) Happy

A smile expression is an expression that can show that someone is feeling happiness or liking
something. The happy expression is on the upward movement of the cheek muscles and the sides
or edges of the lips to form a smile.

Department of ECE,BMSCE 28 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Figure 10. Characteristic of happy expressions

2) Anger

Anger facial expressions arise from the match between what is expected and a reality. The
expression is shown on both sides of the inner eyebrows which are merging and leaning down,
while the lips are narrowing, and the way the eyes are sharp when looking.

Figure 11. Characteristic of anger expression

3) Sadness

A face that shows sadness appears when disappointment or a feeling of missing something.
Based on the characteristics of a sad facial expression when the eye loses focus, the lips are
pulled downwards, and the upper eyelid droops.

Figure 12. Characteristic of sad expression

4) Fear

Department of ECE,BMSCE 29 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

The form of expression that appears when someone experiences an inability to cope with any
event or in a scary atmosphere, then that person is said to be afraid. The expression of fear on a
person’s face is seen from the two eyebrows that rise at the same time, the eyelids tighten, and
the lips that are open horizontally.

Figure 13. Characteristic expression of fear

5) Disgust

A person who expresses his face in a state of disgust due to seeing something not common or
listening to information that is not worth hearing. An expression of disgust will be read when a
person’s face in the area of the nose bridge is wrinkled and the upper lip rises.

Figure 14. Expression of disgust

6) Surprise

Department of ECE,BMSCE 30 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Expressions of surprise are obtained when someone does not know beforehand an event or
message received that is sudden, unexpected or important. Expression is a shocked face
represented by the raised eyebrows, the eyes wide open, and the mouth opening reflflex.

Figure 15. Characteristic expression of surprise

7) Contempt The facial expression of a person who is identifified as being arrogant and has no
sense of respect at the other person often underestimates other people. The expression is shown
from a movement that raises one corner of the lips.

Figure 16. Characteristics of contempt expression

The dataset from a Kaggle Facial Expression Recognition Challenge (FER2013) is used for
the training and testing. It comprises pre-cropped, 48-by-48-pixel grayscale images of faces
each labeled with one of the 7 emotion classes: anger, disgust, fear, happiness, sadness,
surprise, and neutral. Dataset has training set of 35887 facial images with facial expression
labels.. The dataset has class imbalance issue, since some classes have large number of
examples while some has few. The dataset is balanced using oversampling, by increasing

Department of ECE,BMSCE 31 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

numbers in minority classes. The balanced dataset contains 40263 images, from which 29263
images are used for training, 6000 images are used for testing, and 5000 images are used for
validation.
 

Emotion Quantity

Disgust 547

Anger 4953

Surprise 4002

Sadness 6077

Calm 6198

Happiness 8989

Fear 5121

Table 2. FER2013 dataset label breakdown (FER2013)

The following process occurs for each image passed through a neural network during training:
 

1. The neural network makes an emotion prediction based on its current weights and
parameters.
2. The neural network compares the predicted emotion to the true emotion of the image to
calculate a loss value.
3. The loss value is used to adjust the weights of the neural network.
The purpose of the training set is to build the model by adjusting its weights to more accurately
make predictions in the future, with each iteration of the process refining what the network has
‘learned’.

Department of ECE,BMSCE 32 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

The validation set is used to test the neural network after it has been trained. By having a separate
set of sample images in the validation set which was not used for training, we can evaluate the
model more objectively.

Figure 17. Training, Testing and Validation Data distribution (FER2013)

Dataset Challenges:

1. Imbalance: Imbalance is when one class has many more images than another class. This
results in the model being biased towards one class. For example, if there are 2000 images for
the happy expression and 500 images for the fear expression, then the model will be biased

Department of ECE,BMSCE 33 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

towards the happy expression. Data augmentation is done to avoid this problem. Data
augmentation increases the amount of data using techniques like cropping, padding, and
horizontal flipping.
2. Contrast variation: Some images in the dataset can be too dark and some can be too
light. Since images contain visual information, higher contrast images have more information
than lower contrast images. A CNN takes images as input, automatically learns image
features and classifies the images into output classes. Thus, variations in image contrast affect
CNN performance. This problem can be solved by changing the images to focus on the faces.
3. Intra-class variation: Some images in the dataset are not human faces as there are
drawings and animated faces. The features in real and animated faces differ and this creates
confusion when the model is extracting landmark features. Model performance will be better
if all images in the dataset are human faces so other images should be removed.
4. Occlusion: Occlusion is when part of the image is covered. This can occur when a hand
covers a part of the face such as the right eye or nose. A person wearing sunglasses or a mask
also creates occlusion. This indicates that eyes and noses have primary features which are
important to extract and recognize emotions. Thus, 7 occluded images should be removed
from the dataset as the model cannot recognize emotions from these images.

8. SIMULATION RESULTS

Detection of neutral face expression

Department of ECE,BMSCE 34 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Detection of happy face expression

Department of ECE,BMSCE 35 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Detection of angry face expression

Detection of surprised face expression

Department of ECE,BMSCE 36 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Detection of scared face expression

Detection of disgust face expression

Department of ECE,BMSCE 37 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Detection of sad face expression

Detection of angry face expression

Department of ECE,BMSCE 38 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

9. CONCLUSION

Facial emotion recognition using convolutional neural networks (FERC) is a novel way of
facial emotion detection that uses the advantages of CNN and supervised learning (feasible
due to big data). The background removal added a great advantage in accurately determining
the emotions. FERC could be the starting step, for many of the emotion-based applications
such as lie detector and also mood-based learning for students, etc.

This work proposes a convolution neural network model for facial emotion recognition. The
model classifies 7 different facial emotions from the image dataset. The model has
comparable training accuracy and validation accuracy which convey that the model is having
a best fit and is generalized to the data. The work can be extended to find out the changes in
emotion using a video sequence which in turn can be used for different real time applications
such as feedback analysis, etc. This system can also be integrated with other electronic
devices for their effective control.

Department of ECE,BMSCE 39 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

10. FUTURE SCOPE

There are a series of approaches that would either increase the performance of the system or
extent its functionality. This can be done by using different neural networks and latest datasets.
In future this work can be used to develop a Emotion based music system application in which
music is played according to the persons emotion.

Department of ECE,BMSCE 40 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

Department of ECE,BMSCE 41 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

References

1. Akriti Jaiswal, A. Krishnama Raju, Suman Deb, ‘‘Facial emotion detection using deep
learning ”,2020 International Conference for Emerging Technology (INCET), pp. 1-5,
2020.
2. Pranav E,Suraj Kamal, Satheesh Chandran C,Supriya M.H,‘‘Facial Emotion Recognition
Using Deep Convolutional Neural Network”,IEEE Access,2020 6th International
Conference on Advanced Computing and Communication Systems (ICACCS), pp. 317-
320, 2020.
3. Shuang Liu,Dahua Li,Qiang Gao,Yu Song,‘‘Facial Emotion Recognition Based on
CNN”,IEEE Access ,2020 Chinese Automation Congress (CAC), pp. 398-403, 2020.
4. Ninad Mehendale,‘‘Facial emotion recognition using convolution neural
networks”,Springer, Berlin, pp 1–20, (2020).
5. Balaji Balasubramanian,Pranshu Diwan, Rajeshwar Nadar,Anuradha Bhatia,‘‘Analysis of
Facial Emotion Recognition”,2019 3rd International Conference on Trends in Electronics
and Informatics (ICOEI), pp. 945-949, 2019.
6. Charvi Jain, Kshitij Sawant,Mohammed Rehman,Rajesh Kumar,‘‘Emotion Detection and
Characterization using Facial Features’’, 2018 3rd International Conference and
Workshops on Recent Advances and Innovations in Engineering (ICRAIE), pp. 1-6, 2018.
7. Kaustubh Kulkarni, Ciprian Adrian Corneanu, Ikechukwu Ofodile,Sergio Escalera, Xavier
Bar Sylwia Hyniewska,Juri Allik, and Gholamreza Anbarjafari,‘‘Automatic Recognition
of Facial Displays of Unfelt Emotions”,IEEE Transactions on Affective Computing, vol.
12, no. 2, pp. 377-390, 1 April-June 2021.
8. Forugh Mozafari,Hooman Tahayori,‘‘Emotion Detection by Using Similarity
Techniques”,2019 7th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS), pp.
1-5, 2019.
9. Manoj Moolchandani,Shivangi Dwivedi ,Samarth Nigam,Kapil Gupta,‘‘A survey on:
Facial Emotion Recognition and Classification”,2021 5th International Conference on
Computing Methodologies and Communication (ICCMC), pp. 1677-1686, 2021.
10. Ming Li,Hao Xu,Xingchang Huang, Zhanmei Song, Xiaolin Liu, and Xin Li, ‘‘Facial
Expression Recognition with Identity and Emotion Joint Learning”, IEEE Transactions on
Affective Computing, vol. 12, no. 2, pp. 544-550, 1 April-June 2021.

Department of ECE,BMSCE 42 | P a g e
 Facial Sentiment Analysis Using Affective
Computing

11. Ma Xiaoxi, Lin Weisi, Huang Dongyan, Dong Minghui, Haizhou Li, “Facial Emotion
Recognition”,IEEE International Conference,2017.
12. Fathimath Hafisa Hariz, K Nithin Upadhyaya, Sumayya, T F Mohammad Danish,
Bharatesh b, Shashank M Gowda, “Facial emotion recognition for smart
applications”,IEEE Access NCCDS - 2020 Conference Proceedings.
13. Dr. V. Umarani, Mungara Srilakshmi,“Face Emotion Recognition: A Brief Review” 2021
volume 9 International Journal of Creative Research Thoughts (IJCRT).
14. Ram Kumar Madupu, Chiranjeevi Kothapalli, Vasanthi Yarra, Sonti Harika, CM AK
Zeelan Basha, “Automatic human emotion recognition system using facial expression with
convolution neura network”,2020 4th International Conference on Electronics,
Communication and Aerospace Technology (ICECA), pp. 1179-1183,2020.
15. Sabrina Begaj, Ali Osman Topal, Maaruf Ali, “Emotion recognition based on facial
expressions using cnn”,2020 International Conference on
Computing, Networking, Telecommunications & Engineering Sciences Applications
(CoNTESA), pp. 58-63, 2020.
16. Ansamma John, Abhishek MC, Ananthu S Ajayan, S Sanoop, Vishnu R Kumar,“Real-
time facial emotion recognition system with improved preprocessing and feature
extraction”,2020 Third International Conference on Smart Systems and Inventive
Technology (ICSSIT), pp. 1328-1333, 2020.
17. A. Amini, A. Soleimany, MIT deep learning open access course 6.S191, available online:

http://introtodeeplearning.com/ (2020).

18. C. Pramerdorfer and M. Kampel, “Facial expression recognition using convolutional neural
networks: state of the art,” arXiv preprint arXiv:1612.02903, 2016.

19. https://iq.opengenus.org/inception-v3-model-architecture/.

Department of ECE,BMSCE 43 | P a g e
 Facial Sentiment Analysis Using
Affective Computing
APPENDIX A:
Plagiarism Report

Department of ECE,BMSCE 44 | P a g e
 Facial Sentiment Analysis Using
Affective Computing

APPENDIX B:
Program Outcomes (Pos) of the M. Tech Program:

Department of ECE,BMSCE 45 | P a g e
 Facial Sentiment Analysis Using
Affective Computing

APPENDIX C:
Research Publications

Department of ECE,BMSCE 46 | P a g e