1

Chapter 7 Analysis of Variance (ANOVA)

[See: Design & Analysis of Experiments (Book Ch. 15)]

7.2 Development of the ANOVA Problem and Solution

[See the textbook section 15.2 ONE-WAY DESIGNS]

Consider the random vector X  [ X 1 ,  , X m ] ~ N (  , ) . Express the mean as

tr

  1    1  [1 , ,  m ]tr . (2.1)

Definition 2.1 In (1), the constant μ is referred to as the grand mean, and the elements of  are
referred to as the treatment effects.

PROBLEM 1. Test H o :   0 versus H 1 :   0 .

The decision rule for this hypothesis test is not easy to construct, unless we make a number of
simplifying assumptions (that may or may not, in fact, be true). The first is:

Assumption (A1):    2  .

It follows that we can express X as:

X    where E ( )  0 and Cov( )   2  . (2.2).

In other words, X  [ X 1 , , X m ] ~ N (  ,   I ) .
tr 2

The method known as ANOVA (Analysis Of VAriance) is a standard method for solving
PROBLEM 1. The solution also requires a second assumption:
n
Assumption (A2): The data collection random variables { X k }k 1 associated with
X  [ X 1 , , X m ]tr ~ N (  ,  2 I ) are also assumed to be mutually independent.

Hence, in relation to PROBLEM 1, suppose that, in fact, H0 is true. We then have:

{X k j } j 1,m ~ iid N (  ,  2
)
k 1,n

Before we proceed to the heart of ANOVA it helps to have some convenient notation:
m
 x x   x k2 . This quantity is called the squared norm
2 tr
Definition 2.2 For x   ,
m
x
k 1

associated with the m-D array of numbers x  ( x1 , x2 ,, xm ) .

tr
 2

Theorem 2.1 [Similar to THEOREM 15.1 on p.489]

2 2 2
E X  1  E    (2.3)

where, from (2.2),   X  .

Proof: The proof is given here to demonstrate the value of the linearity of E(*), along with a
common ‘trick’ that is used (i.e. adding and subtracting the same thing).
2 2 2
E X  1  E X  ( 1   )    E    E[(   ) tr (   )]

2 2 2 2
E   2 E ( )   E   
tr
▄

The reader should compare the above proof to that given on p.489 of the textbook. The latter
proof is given in terms of numbers and estimates. The above proof is in terms of random
variables and their true (not estimated) means. Hence, the two theorems are not exactly the same.
Nonetheless, the above theorem not only takes great advantage of the linearity of E(*), but it also
provides clearer insight as to how the various ‘sums of squares’ in ANOVA relate to the total
‘sum of squares’, as are now defined.

Definition 2.3 The term on the left side of (2.3) is called the Total Mean Squared Error
(TMSE). The left term on the right side of (2.3) is called the Total Error Mean Squared Error
(TEMSE). And the right term on the right side of (2.3) is called the Total Treatment Squared
Error (TTSE).

It is now a simple task to obtain the form of Theorem 2.1 above and Theorem 15.1 (p.489).
n
Consider the collection of random variables { X j } j 1 ~ iid f X ( x ) . Use this collection, and
replace expected values by their approximants; namely averages:
2 2 2
In the equation E X  1  E    we obtain the following approximations:
 
  x ;  i  xi  x ; hence for fixed i :  ij  xij  xi . Thus,

1 n 1 n tr  tr 

n j 1
( x j  x ) ( x  x )    j  j    . or
tr

n j 1

THEOREM 15.1 (Book p.489)

m n m n m

 ( x
i 1 j 1
ij  x )  ( xij  xi )  n ( xi  x ) 2 .
2

i 1 j 1
2

i 1
(2.4)
 3

Remark. Replacing quantities by their moment estimators does not guarantee the equality in
(2.4). This equality holds in this particular instance, as is proven in the book.

The reason for presenting THEOREM 2.1 was to offer twofold motivation. First, it offered clear
insight into the variability associated with the TMSE. Second, it provided the motivation to use
moment estimators to arrive at THEOREM 15.1. In this way, we are guided to use the test
statistic on the left side of (2.4) in relation to PROBLEM 1. Before we address the distribution of
this statistic [of course, this requires x’s to be replaced by X’s in (2.4)], it is appropriate to take
advantage of the simplicity of (2.3) in order to gain some insight into the behavior of this
hypothesis testing approach.

Properties of (2.3) in Relation to PROBLEM 1:

(P1) Assuming H o is true, then the TTSE is zero, and (2.3) is an identity.

(P2) Assuming H o is false, then the TMSE must be greater than if it were true. This observation
leads immediately to the form of the decision rule: If the test statistic exceeds a given threshold,
we announce H 1 .

(P3) Since the TTSE is the squared norm of  , there is no way to incorporate any prior
information about this parameter into the test. For example, the following two extremely different
 structures would yield similar test results:

  [1, 1, 1, 1]tr ;   [ 2, 0, 0, 0]tr .

The insight (P3) provided by (2.3) would suggest that, if one did in fact have some prior
knowledge of how the components of  were distributed, then this knowledge could be used to
assign a prior pdf for it. This is, in a sense, the essence of Bayesian Estimation Theory.

Development of the Most Appropriate Test Statistic for PROBLEM 1-

We alluded to a test statistic in (P2) above. Here, we develop the standard test statistic associated
with PROBLEM 1. To this end, we express (4) in its random variable form:

m n m n m

 ( X
i 1 j 1
ij  X  )  ( X ij  X i )  n ( X i  X  ) 2
2

i 1 j 1
2

i 1

SST = SSE + SS(Tr) (5)

where

SST = Sum of Squares- Total

SSE = Sum of Squares- Error

SS(Tr) = Sum of Squares- Treatment

 4

Assumption: H o is true.

Note that for each i,j we have X i , j ~ N (  ,  ) , hence ( X i , j   ) /  ~  1 . Also,

2 2 2 2

X i ~ N (  ,  2 / n) , hence ( X i   ) 2 /( 2 / n) ~ 12 . Thus, we obtain the following:

m n
1 1
SST:
2
 ( X
i 1 j 1
ij   ) 2 ~  mn
2

2
SST ~  mn
2
1 (6a)

m n
1 1
SSE:
2
 ( X
i 1 j 1
ij   i ) 2 ~  mn
2

2
SSE ~  m2 ( n 1) (6b)

1 m 1
2
SS(Tr): ( X i   ) 2 ~  m2  2 SS (Tr ) ~  m2 1 (6c)
 i1 
Remark. Recall that, if random variables U ~  u and V ~  v are independent, then
2 2

U  V ~  u2 v . Even though the distribution (6a) happens to correspond to the distribution of
(6b) + (6c), this may not imply that (6b) and (6c) are independent, since the above relation is not
if and only if.

There are a variety of decision rules that can be constructed using the statistics in (6). The most
common derived test statistic is based on the following result:

Result 1 (see Book Theorem 8.14). If U ~  u and V ~  v are independent, then

2 2

U /u
~ F (u , v )
V /v

From this result, we obtain the most commonly used test statistic related to PROBLEM 1:

SS (Tr ) /( m  1)  MS (Tr )
F   ~ F [ m  1, m( n  1)] . (7)
SSE /(m( n  1) MSE

We now repeat PROBLEM 1, and provide the solution to it:

PROBLEM 1. Consider the random vector X  [ X 1 ,  , X m ] ~ N (  , ) . Express the mean

tr

as         [ 1 , ,  m ] . Then to conduct the test of

tr

H o :   0 versus H 1 :   0

we use (7) as our test statistic, along with the corresponding

Decision Rule: If F  f m 1,m ( n 1) ( ) we announce H o with false alarm probability δ.

▄
 5

Example 1.1 [Textbook Problem 15.16 on p.513]

To compare the effectiveness of three types of coatings on instrument panel dials, a total
of 24 dials (8 dials for each type of coating) were tested. Specifically, they were
illuminated by ultraviolet light. When the light was removed, the time for the glow to
disappear was measured. The test results are given in the following Matlab code:
%PROGRAM NAME: example7_1_1.m
x1=[52.9 62.1 57.4 50.0 59.3 61.2 60.8 53.1]';
x2=[58.4 55.0 59.8 62.5 64.7 59.9 54.7 58.4]';
x3=[71.3 66.6 63.4 64.7 75.8 65.6 72.9 67.3]';
m = 3; n = 8; nm = n*m;
x = [x1 x2 x3];
%=====================
xvec = [x1 ; x2 ; x3];
mxdotdot = mean(xvec);
SST = sum((xvec-mxdotdot).^2);
%-------------------
mxdot = mean(x);
SSTr = n*sum((mxdot - mxdotdot).^2);
%-------------------
xdotvec = [x1-mxdot(1) ; x2-mxdot(2) ; x3-mxdot(3)];
SSE = sum(xdotvec.^2);
mxdotdot
mxdot'
[SST SSTr SSE]
%--------------------
pause
MSTr = SSTr/(m-1);
MSE = SSE/(m*(n-1));
f = MSTr/MSE
pause
%==========================
% Test muD=mu2-mu1 = 0 versus muD>0
d = x2 - x1;
md = mean(d);
stdd = std(d)/8^.5;
t = md/stdd
tth = tinv(.95,7)

Running the above code gave:

mxdotdot = 61.5750 & mxdot= [57.1000 59.1750 68.4500]
[SST SSTr SSE] = [ 944.4250 584.4100 360.0150]
f =17.0446
Now consider H0: All 3 true mean values are equal versus H1: Not all equal, with a false
alarm probability (i.e. significance level) 0.01.

Our decision rule is: If Fm1,n ( m1)  F2, 21  f th we will announce H1.
 6

Since finv(.99,2,21) = 5.7804 is less than 17.0446 we will announce H1.

A closer look at the sample means shows what should have been obvious from the raw
data; namely that the type-3 coating performs better than the others.
now let’s investigate the performance of the Type-1 in relation to the Type-2 coatings. To
this end, let    2  1 , and consider the hypothesis test H 0 :   0 versus H1 :   0 with
a false alarm probability of 0.05.
   1 8 1 8
Then    2  1   ( X k1  X k 2 )   Dk
8 k 1 8 k 1

where the generic random variable D  X 2  X 1 ~ N (  D   2  1 ,  D2   12   22 ) .


 0
And so our test statistic is: T    ~ t7 . From the lower portion of the above code, we


find that t =0.8900 and tth = 1.8946. Hence, we will announce H0. □

_________________________________
The following pages were taken from the internet. They include a reasonably good
discussion of ANOVA from a statistician’s perspective. They also include a number
of practice examples. More of the same are given in Chapter 15 of the book.
7
 8

The following discussion was obtained off the internet. It is included to give you an
idea of how ANOVA is typically approached. It also includes examples.
One Way ANOVA
Analysis of Variance (ANOVA)
ANOVA is the statistical procedure for determining whether significant
differences exist in an experiment containing two or more sample means.
ANOVA may be used for interval or ratio data.
Example: A researcher wants to test for the effectiveness of drug and counseling
therapies for the treatment of depression. He randomly assigns clinically
depressed subjects to one of 5 groups and measures their level of depression
after 2 months. The five groups are a no intervention control group, a placebo
drug control group, a drug only experimental group, a counseling only
experimental group, and a drug and counseling experimental group.
Why not use t-tests to compare the mean level of depression for each of these
five groups?
We would have to use a separate test for every pair of means. We would need
ten tests (A&B, A&C, A&D, A&E, B&C, B&D, B&E, C&D, C&E, and D&E). If we
set a significance level of .05 for each test and run ten tests in this one
experiment, the error possible in this experiment (called experiment-wise error)
would be very high. It would be one minus the total probability of not committing
an error. The probability of not committing an error is multiplied with every test
(.9510 = .6). So the experiment-wise error would be 1 -.95 10 = .4. There would be
up to a 40% chance of committing a type one error!
The solution to this problem of multiplying experiment-wise error is to use
ANOVA to test the overall effect of the treatment. For ANOVA we calculate an F-
score and use the F-distribution to help decide whether to reject the null
hypothesis or not.
Assumptions
1. random samples, interval or ratio scores
2. normal distribution
 9

3. homogeneity of variance

Hypotheses look like this:

Ho: 1 = 2 = 3 = 4 =  = k

Ha: Not all 's are the same.

k represents the number of groups you are comparing.
Idea behind ANOVA
All groups of scores are going to have some variance associated with it.
Not everyone's score is the same. For each particular kind of
measurement there is an associated amount of variance. All the people in
the group are going to vary somewhat in a particular population (within-
group variance). Two or more different populations are going to vary by
similar amounts if there is homogeneity of variance (one of our
assumptions). But two or more different populations may vary a lot from
each other (between-group variance).
 10

So, ANOVA allows us to take apart this variability (variance) of all the
scores and see how much of the variability is from within the groups
(within-group variance) and how much is due to the fact that we have
different groups with different population means (between-group
variance).
Total Variance = Between Group Variance + Within Group Variance
 11

 
 
If there is a large amount of the total variance that is due to differences
between the groups, then we will conclude that our means for those
sample groups came from different populations.
Like for the t-tests where we don't know the population variance we have
to estimate it.
The Definitional formula for Estimated Population Variance is:

Remember that the numerator of this equation is the sum of each of the
deviations from the mean squared. We abbreviate this and call it the Sum
of Squares (SS). The definitional formula for variance takes this Sum of
Squares and divides it by the number of subjects (less one when we are
estimating because we have one less degree of freedom). When we
divide a sum by the number of items in that sum we usually call this the
mean. Therefore, the definitional formula for variance can also be referred
to as the Mean of the Sum of Squares. We abbreviate this and call it the
Mean of Squares (MS).
 12

So, estimated population variance is MS = SS/df

Remember that we were going to partition this variance (MS) into two
parts; the part due to variance within the groups (MS wn) and the part due to
variance between the groups (MSbn).
MSwn is an estimate of the population error (variance that cannot be
explained by the independent variable). It is the average variability of the
scores in each group around the mean of that group. This is the variability
that is due to individual differences, not due to the treatment.

MSwn estimates the population error variance (error)

Sample Estimates Population

MSwn  error

 
MSbn is an estimate of both error variance and the treatment variance
(effect from the independent variable). It is the average variability of the
mean of each group around the grand mean of the entire sample. This is
the variability that is due to individual differences and due to the
treatment.error + treat)

Sample Estimates Population

MSbn  error + treat

 13

 
The F that we calculate is a ratio of these errors. F = MS bn/ MSwn

Sample Estimates Population

F = MSbn  error + treat

MSwn  error

 
If Ho is true. Then the treatment has no effect. The variability between
groups will be the same as the variability within groups.

If, like in these graphs, the groups all have means of about 5.5 and the
scores vary from 4 to 7, then the total variability (variability due to
 14

treatment plus error from individual differences) will be the same as the
variability due to error (individual differences). None of the error variability
is due to the treatment (treat = 0)

Sample Estimates Population

F = MSbn  error + 

MSwn  error .

 
If Ho is false. Then the treatment has some effect. The variability between
groups will greater than the variability within groups.
 
 15

If, like in these graphs, some of the groups have different means and the
scores vary from 3 to 9, then the total variability (variability due to
treatment plus error from individual differences) will be greater than the
variability due to error (individual differences). Some of the variability is
due to the treatment (treat = some amount)

Sample Estimates Population

F = MSbn  error + some amount of treat

MSwn  error .

 
 16

When Ho is false, Ha is true, MSbn is larger than MSwn and Fobt is greater
than 1.

Calculating F

In order to keep all these calculations and results organized, we use the
Analysis of Variance Summary Table
Summary Table of One-Way ANOVA

Source Sum of Squares (SS) df Mean Square (MS) F

Between SSbn dfbn MSbn Fobt

Within SSwn dfwn MSwn  

Total SStot dftot    

Conducting the F-Test

F-obtained is tested against the F-critical from the F-table (pages 495-
497). There are three things you need to look up the F-Critical.
 17

1. df for within variance (left side column)

2. df for between variance (top row)

3. 
(.05 bold numbers on top, .01 not-bold numbers on the bottom)
F-values are always positive since we are dealing with variance (numbers that
have been squared).
Compare your F-obtained to F-critical.
If your F-obtained is bigger than the F-critical then you can reject the null
hypothesis and conclude that there is a significant treatment effect.
If your F-obtained is smaller than the F-critical then you fail to reject the null
hypothesis.
Report the F-test results

F(dfbn, dfwn) = ________, p < (or > if smaller than Fcrit) .

Graph the results (see example below).

Example 1
Suppose a psychologist examined learning performance under three temperature
conditions: (1) 50 , (2) 70 , (3) 90 . The subjects were randomly assigned to
one of three treatment groups and learning was measured on a 7 point scale.
The data are shown below. What should the psychologist conclude about the
effect of temperature on learning? ( = .05)

Group # Learning X2
(x)

1 2 4

1 2 4

1 3 9

1 1 1

1 2 4
 18

2 4 16

2 3 9

2 6 36

2 5 25

2 7 49

3 1 1

3 2 4

3 2 4

3 3 9

3 2 2

Ho:
Ha:
ANOVA Summary Table

Source Sum of Squares df Mean Square (MS) F

(SS)

Between SSbn dfbn MSbn Fobt

Within SSwn dfwn MSwn  

Total SStot dftot    

 19

Fcrit ( , ) =
Answer: F( , ) = ________, p .05
Conclusion:
Graph:

Proportion of Variance Accounted For in the Sample (Effect Size)

SSbn
 =
2

SStot

(2 is called eta squared)

In our example 2 =
Estimate of the Proportion of Variance Accounted For in the Population
(Effect Size)

SSbn - (dfbn)(MSwn)
 =2

SStot + MSwn

(2 is called omega squared)

In our example 2 =

Example 2
A pharmaceutical company has developed a drug that is expected to reduce
hunger. To test the drug, 17 rats were randomly assigned to one of three
conditions. The first sample received the drug every day, the second was given
the drug once a week, and the third sample received no drug at all. The amount
of food eaten by each rat over a one month period was measured. Based on the
following data can you conclude that the drug effects food intake? ( = .05)

Group # Food X2
Intake
(x)
 20

1 2 4

1 4 16

1 1 1

1 1 1

1 2 4

1 3 9

2 4 16

2 3 9

2 6 36

2 10 100

2 7 49

2 5 25

3 11 121

3 12 144

3 8 64

3 5 25

3 6 36
 21

Ho:
Ha:
ANOVA Summary Table

Source Sum of Squares df Mean Square (MS) F

(SS)

Between SSbn dfbn MSbn Fobt

Within SSwn dfwn MSwn  

Total SStot dftot    

Fcrit ( , ) =
Answer: F( , ) = ________, p .05
Conclusion:
Graph:
Effect size for the sample:
Estimated effect size for the population:

Post-hoc Comparisons
1. Fisher's protected t-test (for unequal n's)
Protects the experiment-wise error rate, so that the probability of a type I
error for all the comparison's together is <.05.
Notice that the pooled variance is used in the denominator. The t-test
procedure is the same as the one's that we have done previously. Always
use a two-tailed test.
 22

2. Tukey's HSD (Honestly Significant Difference)multiple comparisons test:

Used only when the n's in all levels of the factor are equal.
Steps:

1. Find qk, using the q-table on pages 498-499. You will need  (.05
bold, .01 not-bold), k (the number of means you are comparing, top
row), and dfwn (left column).
2. Compute the HSD, (the minimum difference between any two
means that is required for them to be considered significantly
different).

3. Determine which pairs are significantly different (subtract one mean

from the other and compare it to HSD, if the difference between
your means is bigger than HSD, then the difference is significant.)
Use a table to make these comparisons clear (see example below).
Which post-hoc test would you use for example 1?

  x1 x2 x3

     

x1        

x2        

x3        
 23

 
Which post-hoc test would you use for example 2?

  x1 x2 x3

     

x1        

x2        

x3        

Power
Anything that increases Fobt will increase power. That means that having larger
differences between the means (MSbn is larger) and decreasing the variability of
the scores within conditions (MSwn is smaller) will increase power. Maximizing n
in each condition, which increases dfwn will also minimize MSwn and increase
power.