UNIT 3: Association Rules and Regression

I]Apriori Algorithm

With the quick growth in e-commerce applications, there is an accumulation vast quantity of data
in months not in years. Data Mining, also known as Knowledge Discovery in Databases(KDD),
to find anomalies, correlations, patterns, and trends to predict outcomes.

Apriori algorithm is a classical algorithm in data mining. It is used for mining frequent itemsets
and relevant association rules. It is devised to operate on a database containing a lot of
transactions, for instance, items brought by customers in a store.

It is very important for effective Market Basket Analysis and it helps the customers in
purchasing their items with more ease which increases the sales of the markets. It has also been
used in the field of healthcare for the detection of adverse drug reactions. It produces association
rules that indicates what all combinations of medications and patient characteristics lead to
ADRs.

Association rules

Let be a set of n attributes called items and be the set of transactions. It is called database. Every
transaction, in has a unique transaction ID, and it consists of a subset of itemsets in .

A rule can be defined as an implication, where and are subsets of , and they have no element in
common, i.e., . and are the antecedent and the consequent of the rule, respectively.

Let‟s take an easy example from the supermarket sphere. The example that we are considering is
quite small and in practical situations, datasets contain millions or billions of transactions. The
set of itemsets, ={Onion, Burger, Potato, Milk, Beer} and a database consisting of six
transactions. Each transaction is a tuple of 0's and 1's where 0 represents the absence of an item
and 1 the presence.

An example for a rule in this scenario would be {Onion, Potato} => {Burger}, which means that
if onion and potato are bought, customers also buy a burger.

Transaction ID Onion Potato Burger Milk Beer

1 1 1 0 0

0 1 1 1 0

0 0 0 1 1

1 1 0 1 0
 1 1 1 0 1

1 1 1 1 1

There are multiple rules possible even from a very small database, so in order to select the
interesting ones, we use constraints on various measures of interest and significance. We will
look at some of these useful measures such as support, confidence, lift and conviction.

Support

The support of an itemset , is the proportion of transaction in the database in which the item X
appears. It signifies the popularity of an itemset.

In the example above, .

If the sales of a particular product (item) above a certain proportion have a meaningful effect on
profits, that proportion can be considered as the support threshold. Furthermore, we can identify
itemsets that have support values beyond this threshold as significant itemsets.

Confidence

Confidence of a rule is defined as follows:

It signifies the likelihood of item Y being purchased when item X is purchased. So, for the rule
{Onion, Potato} => {Burger},

This implies that for 75% of the transactions containing onion and potatoes, the rule is correct. It
can also be interpreted as the conditional probability , i.e, the probability of finding the itemset
in transactions given the transaction already contains .

It can give some important insights, but it also has a major drawback. It only takes into account
the popularity of the itemset and not the popularity of . If is equally popular as then there will
be a higher probability that a transaction containing will also contain thus increasing the
confidence. To overcome this drawback there is another measure called lift.

Lift

The lift of a rule is defined as:

This signifies the likelihood of the itemset being purchased when item is purchased while taking
into account the popularity of .

In our example above,

If the value of lift is greater than 1, it means that the itemset is likely to be bought with itemset ,
while a value less than 1 implies that itemset is unlikely to be bought if the itemset is bought.
Conviction

The conviction of a rule can be defined as:

For the rule {onion, potato}=>{burger}

The conviction value of 1.32 means that the rule {onion,potato}=>{burger} would be incorrect
32% more often if the association between and was an accidental chance.

How does Apriori algorithm work?

So far, we learned what the Apriori algorithm is and why is important to learn it.

A key concept in Apriori algorithm is the anti-monotonicity of the support measure. It assumes
that

All subsets of a frequent itemset must be frequent

Similarly, for any infrequent itemset, all its supersets must be infrequent too

Let us now look at the intuitive explanation of the algorithm with the help of the example we
used above. Before beginning the process, let us set the support threshold to 50%, i.e. only those
items are significant for which support is more than 50%.

Step 1: Create a frequency table of all the items that occur in all the transactions. For our case:

Item Frequency (No. of transactions)

Onion(O) 4

Potato(P) 5

Burger(B) 4

Milk(M) 4

Beer(Be) 2

Step 2: We know that only those elements are significant for which the support is greater than or
equal to the threshold support. Here, support threshold is 50%, hence only those items are
significant which occur in more than three transactions and such items are Onion(O), Potato(P),
Burger(B), and Milk(M). Therefore, we are left with:

Item Frequency (No. of transactions)

Onion(O) 4

Potato(P) 5
Burger(B) 4

Milk(M) 4

The table above represents the single items that are purchased by the customers frequently.

Step 3: The next step is to make all the possible pairs of the significant items keeping in mind
that the order doesn‟t matter, i.e., AB is same as BA. To do this, take the first item and pair it
with all the others such as OP, OB, OM. Similarly, consider the second item and pair it with
preceding items, i.e., PB, PM. We are only considering the preceding items because PO (same as
OP) already exists. So, all the pairs in our example are OP, OB, OM, PB, PM, BM.

Step 4: We will now count the occurrences of each pair in all the transactions.

Itemset Frequency (No. of transactions)

OP 4

OB 3

OM 2

PB 4

PM 3

BM 2

Step 5: Again only those itemsets are significant which cross the support threshold, and those are
OP, OB, PB, and PM.

Step 6: Now let‟s say we would like to look for a set of three items that are purchased together.
We will use the itemsets found in step 5 and create a set of 3 items.

To create a set of 3 items another rule, called self-join is required. It says that from the item pairs
OP, OB, PB and PM we look for two pairs with the identical first letter and so we get

OP and OB, this gives OPB

PB and PM, this gives PBM

Next, we find the frequency for these two itemsets.

Itemset Frequency (No. of transactions)

OPB 4

PBM 3
Applying the threshold rule again, we find that OPB is the only significant itemset.

Therefore, the set of 3 items that was purchased most frequently is OPB.

The example that we considered was a fairly simple one and mining the frequent itemsets
stopped at 3 items but in practice, there are dozens of items and this process could continue to
many items. Suppose we got the significant sets with 3 items as OPQ, OPR, OQR, OQS and
PQR and now we want to generate the set of 4 items. For this, we will look at the sets which
have first two alphabets common, i.e,

OPQ and OPR gives OPQR

OQR and OQS gives OQRS

In general, we have to look for sets which only differ in their last letter/item.

Now that we have looked at an example of the functionality of Apriori Algorithm, let us
formulate the general process.

General Process of the Apriori algorithm

The entire algorithm can be divided into two steps:

Step 1: Apply minimum support to find all the frequent sets with k items in a database.

Step 2: Use the self-join rule to find the frequent sets with k+1 items with the help of frequent k-
itemsets. Repeat this process from k=1 to the point when we are unable to apply the self-join
rule.

This approach of extending a frequent itemset one at a time is called the “bottom up” approach.

Mining Association Rules

Till now, we have looked at the Apriori algorithm with respect to frequent itemset generation.
There is another task for which we can use this algorithm, i.e., finding association rules
efficiently.

For finding association rules, we need to find all rules having support greater than the threshold
support and confidence greater than the threshold confidence.

But, how do we find these? One possible way is brute force, i.e., to list all the possible
association rules and calculate the support and confidence for each rule. Then eliminate the rules
that fail the threshold support and confidence. But it is computationally very heavy and
prohibitive as the number of all the possible association rules increase exponentially with the
number of items.
Given there are n items in the set , the total number of possible association rules is .

We can also use another way, which is called the two-step approach, to find the efficient
association rules.

The two-step approach is:

Step 1: Frequent itemset generation: Find all itemsets for which the support is greater than the
threshold support following the process we have already seen earlier in this article.

Step 2: Rule generation: Create rules from each frequent itemset using the binary partition of
frequent itemsets and look for the ones with high confidence. These rules are called candidate
rules.

Let us look at our previous example to get an efficient association rule. We found that OPB was
the frequent itemset. So for this problem, step 1 is already done. So, let‟ see step 2. All the
possible rules using OPB are:

OPB, OBP, PBO, B OP, POB, OPB

If is a frequent itemset with k elements, then there are candidate association rules.

We will not go deeper into the theory of the Apriori algorithm for rule generation.

Pros of the Apriori algorithm

 It is an easy-to-implement and easy-to-understand algorithm.

 It can be used on large itemsets.
 Cons of the Apriori Algorithm
 Sometimes, it may need to find a large number of candidate rules which can be
computationally expensive.
 Calculating support is also expensive because it has to go through the entire database.

II] Simple Linear Regression (SLR)

Simple linear regression is a method that enables you to determine the relationship between a
continuous process output (Y) and one factor (X). The relationship is typically expressed in
terms of a mathematical equation such as Y = b + mX
Suppose we believe that the value of y tends to increase or decrease in a linear manner as x
increases. Then we could select a model relating y to x by drawing a line which is well fitted to a
given data set. Such a deterministic model – one that does not allow for errors of prediction –
might be adequate if all of the data points fell on the fitted line. However, you can see that this
idealistic situation will not occur for the data of Table 11.1 and 11.2. No matter how you draw a
line through the points in Figure 11.2 and Figure 11.3, at least some of points will deviate
substantially from the fitted line.
The solution to the proceeding problem is to construct a probabilistic model relating y to x- one
that knowledge the random variation of the data points about a line. One type of probabilistic
model, a simple linear regression model, makes assumption that the mean value of y for a given
value of x graphs as straight line and that points deviate about this line of means by a random
amount equal to e, i.e.

y = A + B x + e,
where A and B are unknown parameters of the deterministic (nonrandom ) portion of the model.
If we suppose that the points deviate above or below the line of means and with expected value
E(e) = 0 then the mean value of y is
y = A + B x.
Therefore, the mean value of y for a given value of x, represented by the symbol E(y) graphs as
straight line with y-intercept A and slope B.

In simple words linear regression is predicting the value of a variable Y(dependent variable)
based on some variable X(independent variable) provided there is a linear relationship between
X and Y.

This linear relationship between the 2 variables can be represented by a straight line
(called regression line).

Now to determine if there is a linear relationship between 2 variables we can simply plot a
scatter plot of variable Y with variable X .If the plotted points are randomly scattered that it can
be inferred that the variables are not related.
There is a linear relationship between the variables.

There is no linear relationship between the variables.

When regression line is drawn some points will lie on the regression line other points will lie in
the close vicinity of it. This is because our regression line is a probabilistic model and our
prediction is approximate. So there will be some errors/deviations from actual/observed value of
variable Y.

But when the linear relationship exist between X and Y we can plot more than one line through
these points. Now how do we know which one is the best fit?
To help us choose the best line we use the concept of “least squares”.

Least Squares
Y=b0 + b1X+e

This the mathematical representation for the regression line where

Y-Dependant variable.

X-Independent variable.

b0 –intercept of the regression line.

b1-slope of the regression line.

e- error/deviation from actual/observed value of variable Y.

Suppose we fit n points of the form (x1,y1) ,(x2,y2)…..(xn,yn)to the above regression line then

Where ei is the difference between ith observed response value and the ith response value that is
predicted by our regression line.

Our aim here is to minimize this error so that we can get the best possible regression line.

Now this error ei can be positive or negative but we are only interested in the magnitude of the
error and not in its sign. Hence we square the errors and minimize the sum of squared
errors(SSE).
(In the above graph the green line is the best fit.)

How do we minimize the sum of squared errors(SSE)?

Remember that b1 and b0 are still unknown to us.

In the least square approach we minimize sum of squared errors(SSE) by choosing the value of
b1 and b0 to be (not diving into math of it)

Multiple Linear Regression (MLR)

This procedure performs linear regression on the selected dataset. This fits a linear model of the
form
Y= b 0 + b 1 X 1 + b 2 X 2 + .... + b k X k + e
where Y is the dependent variable (response) and X 1 , X 2 ,.. .,X k are the independent variables
(predictors) and e is random error. b 0 , b 1 , b 2 , .... b k are known as the regression
coefficients, which have to be estimated from the data. The multiple linear regression algorithm
in XLMiner chooses regression coefficients so as to minimize the difference between predicted
values and actual values.
Linear regression is performed either to predict the response variable based on the predictor
variables, or to study the relationship between the response variable and predictor variables. For
example, using linear regression, the crime rate of a state can be explained as a function of other
demographic factors like population, education, male to female ratio etc

III] What is Logistic Regression ?

Logistic Regression is a classification algorithm. It is used to predict a binary outcome (1 / 0,

Yes / No, True / False) given a set of independent variables. To represent binary / categorical
outcome, we use dummy variables. You can also think of logistic regression as a special case of
linear regression when the outcome variable is categorical, where we are using log of odds as
dependent variable. In simple words, it predicts the probability of occurrence of an event by
fitting data to a logit function.

Derivation of Logistic Regression Equation

Logistic Regression is part of a larger class of algorithms known as Generalized Linear Model
(glm). In 1972, Nelder and Wedderburn proposed this model with an effort to provide a means of
using linear regression to the problems which were not directly suited for application of linear
regression. Infact, they proposed a class of different models (linear regression, ANOVA, Poisson
Regression etc) which included logistic regression as a special case.

The fundamental equation of generalized linear model is:

g(E(y)) = α + βx1 + γx2

Here, g() is the link function, E(y) is the expectation of target variable and α + βx1 + γx2 is the
linear predictor ( α,β,γ to be predicted). The role of link function is to „link‟ the expectation of y
to linear predictor.

Important Points
 1. GLM does not assume a linear relationship between dependent and independent
variables. However, it assumes a linear relationship between link function and
independent variables in logit model.

2. The dependent variable need not to be normally distributed.

3. It does not uses OLS (Ordinary Least Square) for parameter estimation. Instead, it uses
maximum likelihood estimation (MLE).

4. Errors need to be independent but not normally distributed.

Let‟s understand it further using an example:

We are provided a sample of 1000 customers. We need to predict the probability whether a
customer will buy (y) a particular magazine or not. As you can see, we‟ve a categorical outcome
variable, we‟ll use logistic regression.

To start with logistic regression, I‟ll first write the simple linear regression equation with
dependent variable enclosed in a link function:

g(y) = βo + β(Age) ---- (a)

Note: For ease of understanding, I‟ve considered „Age‟ as independent variable.

In logistic regression, we are only concerned about the probability of outcome dependent
variable ( success or failure). As described above, g() is the link function. This function is
established using two things: Probability of Success(p) and Probability of Failure(1-p). p should
meet following criteria:

1. It must always be positive (since p >= 0)

2. It must always be less than equals to 1 (since p <= 1)

Now, we‟ll simply satisfy these 2 conditions and get to the core of logistic regression. To
establish link function, we‟ll denote g() with „p‟ initially and eventually end up deriving this
function.

Since probability must always be positive, we‟ll put the linear equation in exponential form. For
any value of slope and dependent variable, exponent of this equation will never be negative.

p = exp(βo + β(Age)) = e^(βo + β(Age)) ------- (b)

To make the probability less than 1, we must divide p by a number greater than p. This can
simply be done by:
p = exp(βo + β(Age)) / exp(βo + β(Age)) + 1 = e^(βo + β(Age)) / e^(βo + β(Age)) + 1 -----
(c)

Using (a), (b) and (c), we can redefine the probability as:

p = e^y/ 1 + e^y --- (d)

where p is the probability of success. This (d) is the Logit Function

If p is the probability of success, 1-p will be the probability of failure which can be written as:

q = 1 - p = 1 - (e^y/ 1 + e^y) --- (e)

where q is the probability of failure

On dividing, (d) / (e), we get,

After taking log on both side, we get,

log(p/1-p) is the link function. Logarithmic transformation on the outcome variable allows us to
model a non-linear association in a linear way.

After substituting value of y, we‟ll get:

This is the equation used in Logistic Regression. Here (p/1-p) is the odd ratio. Whenever the log
of odd ratio is found to be positive, the probability of success is always more than 50%. A typical
logistic model plot is shown below. You can see probability never goes below 0 and above 1.
Performance of Logistic Regression Model

To evaluate the performance of a logistic regression model, we must consider few metrics.
Irrespective of tool (SAS, R, Python) you would work on, always look for:

1. AIC (Akaike Information Criteria) – The analogous metric of adjusted R² in logistic

regression is AIC. AIC is the measure of fit which penalizes model for the number of model
coefficients. Therefore, we always prefer model with minimum AIC value.

2. Null Deviance and Residual Deviance – Null Deviance indicates the response predicted by a
model with nothing but an intercept. Lower the value, better the model. Residual deviance
indicates the response predicted by a model on adding independent variables. Lower the value,
better the model.

3. Confusion Matrix: It is nothing but a tabular representation of Actual vs Predicted values.

This helps us to find the accuracy of the model and avoid overfitting. This is how it looks like:
 Source: (plug – n – score)

You can calculate the accuracy of your model with:

From confusion matrix, Specificity and Sensitivity can be derived as illustrated below:

Specificity and Sensitivity plays a crucial role in deriving ROC curve.

4. ROC Curve: Receiver Operating Characteristic(ROC) summarizes the model‟s performance

by evaluating the trade offs between true positive rate (sensitivity) and false positive rate(1-
specificity). For plotting ROC, it is advisable to assume p > 0.5 since we are more concerned
about success rate. ROC summarizes the predictive power for all possible values of p > 0.5. The
area under curve (AUC), referred to as index of accuracy(A) or concordance index, is a perfect
performance metric for ROC curve. Higher the area under curve, better the prediction power of
the model. Below is a sample ROC curve. The ROC of a perfect predictive model has TP equals
1 and FP equals 0. This curve will touch the top left corner of the graph.

Note: For model performance, you can also consider likelihood function. It is called so, because
it selects the coefficient values which maximizes the likelihood of explaining the observed data.
It indicates goodness of fit as its value approaches one, and a poor fit of the data as its value
approaches zero.

Logistic Regression Model in R

Considering the availability, I‟ve build this model on our practice problem – Dressify data set.
You can download it here. Without going deep into feature engineering, here‟s the script of
simple logistic regression model:

setwd('C:/Users/manish/Desktop/dressdata')

#load data

train <- read.csv('Train_Old.csv')

#create training and validation data from given data

install.packages('caTools')
library(caTools)

set.seed(88)

split <- sample.split(train$Recommended, SplitRatio = 0.75)

#get training and test data

dresstrain <- subset(train, split == TRUE)

dresstest <- subset(train, split == FALSE)

#logistic regression model

model <- glm (Recommended ~ .-ID, data = dresstrain, family = binomial)

summary(model)

predict <- predict(model, type = 'response')

#confusion matrix

table(dresstrain$Recommended, predict > 0.5)

#ROCR Curve

library(ROCR)

ROCRpred <- prediction(predict, dresstrain$Recommended)

ROCRperf <- performance(ROCRpred, 'tpr','fpr')

plot(ROCRperf, colorize = TRUE, text.adj = c(-0.2,1.7))

#plot glm

library(ggplot2)

ggplot(dresstrain, aes(x=Rating, y=Recommended)) + geom_point() +

stat_smooth(method="glm", family="binomial", se=FALSE)

This data require lots of cleaning and feature engineering.

Reference:

1. https://www.analyticsvidhya.com/blog/2015/11/beginners-guide-on-logistic-regression-
in-r/
2. https://towardsdatascience.com/understanding-the-concept-of-simple-linear-regression-
a572087c253
3. https://www.hackerearth.com/blog/machine-learning/beginners-tutorial-apriori-
algorithm-data-mining-r-implementation/