CHAPTER 1

ARMA MODELS

1.1 INTRODUCTION

This project is about the time analysis based model ARMA. Which is a forecasting
model that applies autoregressive analysis (AR) and moving average (MA) methods to
well-behaved time series data. ARMA assumes that the time series is stationary, and if it
fluctuates, it fluctuates uniformly at specific points in time.

ARMA is essential in studying a time series. It is usually utilized in market research for
long-term tracking data research. This model has better and more accurate spectral
resolution and estimation performance than AR or MA model, but it has heavy parameter
estimation. If the model isn't stationary, then you can achieve stationarity by taking a
series of differences.

1.2 HISTORICAL BACKGROUND

The history of ARMA models start with the 1951 thesis of Peter Whittle Hypothesis
testing in Time Series Analysis. This model was later on popularized in the 1970 book by
George E.P Box and Gwilym Jenkins.
This model is basically a tool for understanding and perhaps, predicting the future values.
This was an attempt to describe changes on the time series using a mathematical
approach, hence this model can only be estimated for univariate time series.

Peter Whittle (27 February 1927 – 10 August 2021) born in Wellington, was a New
Zealand mathematician and statistician, specializing in stochastic networks, optimal
 control, time series analysis, stochastic optimization, and he worked in the field of
stochastic mechanics[1].
From 1967 to his 1994 he was Churchill Professor of Mathematics for his research in
Operational at Cambridge University.
His paper Hypothesis testing in Time Series Analysis, generalized World’s
autoregressive expression is for stationary univariate processes to multivariate processes,
and this was published on 1951 as mentioned above.
A summary of Whittle’s paper also appeared as an appendix to second edition of
World’s book on time series analysis.
Whittle was a lecturer at the Statistical Institute in Uppsala before returning to New
Zealand in 1953.

George Edward Pelham Box (18 October 1919 – 28 March 2013) He was born in
Gravesend, Kent, England, and was a British statistician, who worked in the areas of
quality control, time-series analysis, design of experiments, and Bayesian inference.
He has been called "one of the great statistical minds of the 20th century".
From 1948 to 1956, Box worked as a statistician for Imperial Chemical Industries (ICI).
His name is associated with results in statistics such as Box–Jenkins models.
In 1960, Box moved to the University of Wisconsin–Madison to create the Department of
Statistics.
He was appointed Vilas Research Professor of Statistics (the highest honor accorded to
any faculty member at the University of Wisconsin–Madison) in 1980.
Box published books including Statistics for Experimenters (2nd ed., 2005), Time Series

Analysis: Forecasting and Control (4th ed., 2008)

Layout of Project:
Chapter 1 is all about the basic introduction regarding model and the history.
Moving further, the chapter 2 is based on all the terms used in the ARMA model related
to Time series.
Chapter 3 is all about the basic terms used for ARMA models. As there are specific
conditions regarding the stochastic process under which it is used or applicable.

Chapter 4 briefly describe the example with the python codes of how we can run and plot
the commands.
CHAPTER 2

ARMA MODEL

TIME SERIES:
A time series is a collection of observations of well-defined data items obtained
through repeated measurements over time.

Time series analysis is a specific way of analyzing a sequence of data points collected over an
interval of time.

In time series analysis, analysts record data points at consistent intervals over a set period of
time rather than just recording the data points intermittently or randomly.

For example, measuring the value of retail sales each month of the year would comprise a time
series.

COMPONENTS OF A TIME SERIES

In general, a time series is affected by four components, i.e.Trend, seasonal, cyclical and
irregular components.

 Trend
The general tendency of a time series to increase, decrease or stagnate over a long period of time.
In general, a time series is affected by four components, i.e. trend, seasonal, cyclical and
irregular components.

 Seasonal variation

This component explains fluctuations within a year during the season, usually caused by climate
and weather conditions, customs, traditional habits etc.

In general, a time series is affected by four components, i.e. Trend, seasonal, cyclical and
irregular components.

 Cyclical variation
This component describes the medium-term changes caused by circumstances, which repeat in
cycles.
The duration of a cycle extends over longer period of time.
Average weekly cardiovascular mortality in Los Angeles County. There are508 six-day
smoothed averages obtained by filtering daily values over the 10 year period 1970-1979.
 .

 Irregular variation:

Irregular or random variations in a time series are caused by

unpredictable influences, which are not regular and also do not repeat in a particular pattern.
These variations are caused by incidences such as:

War, strike, earthquake, flood, revolution, etc.

There is no defined statistical technique for measuring random fluctuations in a time series.

Combination of Four Components:

Considering the effects of these four components, two different types of models are generally
used for a time series.

1. Additive Model

Y ( t ) =T ( t )+ S ( t ) +C ( t )+ I ( t )

Assumption: These four components are independent of each other.

 2. Multiplicative Model

Y ( t ) =T ( t ) × S ( t ) × C ( t ) × I (t )

Assumption: These four components of a time series are not necessarily independent
and they can affect one another.

Time Series Example: White Noise

 White Noise
A simple time series could be a collection of uncorrelated random variables,
{Wt} with zero mean µ=0 and a finite variance σ2w, denoted as wt ~ wn(0,σ2).

 Gaussian White Noise

A particular useful white noise is Gaussian white noise, wherein
the {Wt} independent normal random variable with mean 0 and the variance same
as above and this is denoted as wt ~iid N (0 , σ 2).

STOCHASTIC PROCESS:
 A set of observed time series is considered to be a sample of the population a stochastic
(random) process is a statistical phenomenon consisting of a collection of random
variables ordered in time.
 The stochastic process is considered to generate the infinite collection (called the
ensemble) of all possible time series that might have been observed.

 Every member of the ensemble is a possible realization of the stochastic process.

 The stochastic process evolves in time according to probabilistic laws.
 The ensemble of a stochastic process is a statistical population.

STATIONARY STOCHASTIC PROCESS:

Stationary stochastic process is one whose ensemble statistics are the same for any value of time.
A time series is said to be stationary if there is no systematic change in mean (no trend).
If there is no systematic change in variance, and if it contains no strictly periodic variations.

STRICTLY STATIONARY:
A stochastic process is strictly stationary if all statistical measures on it are stationary, i.e. do not
depend on t. The distribution of a strictly stationary stochastic process is the same at time t as at
any other time t + τ.
In other words, strict stationarity means that the joint Distribution only depends on the
“difference” h, not the time as:
(t1, t2. . . Tk)

WEAKLY STATIONARY STOCHASTIC PROCESS:

Stochastic process, denoted X (t), is weakly stationary (or second-order stationary) if both of the
following conditions are satisfied:

1. E ( X ( t))=µ
2. cov[X(t), X(t + τ)] = Cxx( τ )
Where cov(X, Y) is defined as E [(X - µ X) (Y - µ Y)]

The first condition states that the expected value of X (t) is equal to the ensemble mean
regardless of t. This condition means that the mean of the process does not depend on time, i.e. it
is stationary.
The second condition states that the auto covariance of X (t) also does not depend on time, only
on time-difference (τ). (When τ = 0, the auto

Covariance reduces to the variance. Therefore, this condition also means that the variance is
stationary.
In other words, a weakly stationary time series {Xt} must have
Three features:
Finite variation, constant first moment, and that the second moment γX (s,t) only depends on |t −
s| and not depends on s or t.s

The auto covariance measures the linear dependence between two points on the same series
observed at different times. Very smooth series exhibit auto covariance functions that stay large
even when the t and s are far apart, whereas choppy series tend to have auto covariance functions
that are nearly zero for large separations.
Strict stationarity does not assume finite variance thus strictly stationary does not necessarily
imply weakly stationary. Processes like i.e. Cauchy is strictly stationary but not weakly
stationary.
A nonlinear function of a strictly stationary time series is still
Strictly stationary, but this is not true for weakly stationary.
• Weak stationarity usually does not imply strict stationarity as
Higher moments of the process may depend on time t.
• If time series {Xt} is Gaussian (i.e. the distribution functions
Of {Xt} are all multivariate Gaussian), then weakly stationary, Also implies strictly stationary.
This is because a multivariate
Gaussian distribution is fully characterized by its first two moments.
A stationary stochastic process is described by its mean value and the distribution around the
mean, and along with this if that distribution is Gaussian and the observed values of the
stochastic process are mutually independent, then the mean and variance are sufficient
descriptors of the stochastic process.
If that distribution is Gaussian and the observed values of the, Stochastic process are mutually
independent, then the mean and variance are sufficient descriptors of the stochastic process.

Procedure for stochastic process:

The procedure of using known data values to fit a time series with suitable model and
estimating the corresponding parameters. It comprises methods that attempt to understand the
nature of the time series and is often useful for future forecasting and simulation.

There are several ways to build time series forecasting models, but we have more focus on
stochastic process.

 We assume a time series can be defined as a collection of random variables

indexed according to the order they are obtained in time, X1, X2, X3, . . . t will
typically be discrete and vary over the integers t = 0, ±1, ±2, . . .
 Note that the collection of random variables {Xt} is referred to as a stochastic
process, while the observed values are referred to as a realization of the stochastic
process.
 A complete description of a time series, observed as a collection of n random
variables at arbitrary time points t1, t2. . . tn, for any positive integer n, is provided
by the joint distribution function.
 evaluated as the probability that the values of the series are jointly less than the n
constants, c1, c2, . . . , cn
i.e., Ft1, t2... tn (c1, c2. . . cn) = Pr(Xt1 ≤ c1, Xt2 ≤ c2, . . . , Xtn ≤ cn).
 Unfortunately, these multidimensional distribution functions cannot usually be
written easily.
Therefore some informative descriptive measures can be useful, such as mean
function and more.

The auto covariance γX (s, t) of stationary time series depends on s and t only through |s − t|,
thus we can rewrite notation s = t + h, where h represents the time shift.

γX (t + h, t) = cov (Xt+h, Xt) = cov(Xh, X0) = γ(h, 0) = γ(h)

Auto covariance Function of Stationary Time Series

γ (h) = cov(Xt+h, Xt) = E[(Xt+h − µ)(Xt − µ)

Another important measure is called partial autocorrelation, which is the correlation between Xs
and Xt with the linear effect of “everything in the middle” removed.
CHAPTER 3

ARMA MODEL

ARMA MODEL:
“The auto regression and moving average (ARMA) models are used in time series
analysis to describe stationary time series .  (ARMA) models provide a parsimonious description
of a (weakly) stationary stochastic process in terms of two polynomials, one for the auto
regression (AR) and the second for the moving average (MA).

 The general ARMA model was described in the 1951 thesis of Peter Whittle, Hypothesis testing
in time series analysis, and it was popularized in the 1970 book by George E. P.
Box and Gwilym Jenkins .

The first of these polynomials is for auto regression, the second for the moving average. Often
this model is referred to as the ARMA (p,q) model; where:

 p is the order of the autoregressive polynomial,

 q Is the order of the moving average polynomial,

The equation is given by:

 X

Where;

φ = the autoregressive model’s parameters,

θ = the moving average model’s parameters.

c = a constant,

Σ = summation notation,

ε = error terms (white noise).

AUTO REGRESSIVE MODEL:

An autoregressive (AR) model is a representation of a type of random process; as such, it is used

to describe certain time-varying processes in nature, economics, etc. The autoregressive model
specifies that the output variable depends linearly on its own previous values and on a stochastic
term (an imperfectly predictable term).

Thus the model is in the form of a stochastic difference equation (or recurrence relation which
should not be confused with differential equation).

Auto regressive model is used in statistics, econometrics and also in signal processing. However,
this shows that we can say a statistical model an autoregressive only if it predicts the future
values based on previous values.

They are widely used in technical analysis to forecast future security prices. Autoregressive
models implicitly assume that the future will resemble the past.

Therefore, they can prove inaccurate under certain market conditions, such as financial crises or
periods of rapid technological change.

 An AR (1) autoregressive process is one in which the current value is based on the
immediately preceding value.
  While an AR (2) process is one in which the current value is based on the previous two
values.
 An AR (0) process is used for white noise and has no dependence between the terms.

 In addition to these variations, there are also many different ways to calculate the
coefficients used in these calculations, such as the least squares method.

GRAPH OF AR(p) PROCESSES:

The simplest AR process is AR (0), which has no dependence between the terms. Only the
error/innovation/noise term contributes to the output of the process, so in the figure, AR (0)
corresponds to white noise.

For an AR (1) process with a positive , only the previous term in the process and the noise
term contribute to the output. If is close to 0, then the process still looks like white noise,
but as approaches 1, the output gets a larger contribution from the previous term relative to
the noise. This results in a "smoothing" or integration of the output, similar to a low pass filter.

For an AR (2) process, the previous two terms and the noise term contribute to the output. If
both and are positive, the output will resemble a low pass filter, with
the high frequency part of the noise decreased. If is positive while
is negative, then the process favors changes in sign between terms of the process. The output
oscillates. This can be likened to edge detection or detection of change in direction.

For example, an investor using an autoregressive model to forecast stock prices would need to
assume that new buyers and sellers of that stock are influenced by recent market transactions
when deciding how much to offer or accept for the security.
The notation AR (p) indicates an autoregressive model of order p. The AR (p) model is
defined as
  

Where are the parameters of the model, and is white noise. This can be
equivalently written using the backshift operator B as

so that, moving the summation term to the left side and using polynomial notation, we have
An autoregressive model can thus be viewed as the output of an all-pole infinite impulse
response filter whose input is white noise. Some parameter constraints are necessary for the
model to remain wide-sense stationary.
For example, processes in the AR (1) model with  

are not stationary. More generally, for an AR(p) model to be wide-sense

stationary, the roots of the polynomial

Must lie outside the unit circle, i.e., each (complex) root Zi must satisfy |z|i> 1.

MOVING – AVERAGE MODEL:

In time series analysis, the moving-average model (MA model), also known as moving-average
process, is a common approach for modeling univariate time series.

Data must be properly formatted for estimation as a time-series. If this is not done, then
depending on your statistical package of choice, either your estimation will fail to execute or you
will receive erroneous output.

ARMA models include some number of lagged error terms from the MA component, which are
inherently unobservable. Consequently these models cannot be estimated using OLS alone,
unlike AR models.

Common criteria used to evaluate ARMA models are the Akaike Information Criterion (AIC)
and the Bayesian Information Criterion (BIC), also referred to as the Schwarz Information
Criterion (SIC).

The moving-average model specifies that the output variable is cross-correlated with a non-
identical to itself random-variable.
Together with the autoregressive (AR) model, the moving-average model is a special case and
key component of the more general ARMA and ARIMA models of time series, which have a
more complicated stochastic structure.

The moving-average model should not be confused with the moving average, a distinct concept
despite some similarities.

Contrary to the AR model, the finite MA model is always stationary.

The notation MA(q) refers to the moving average model of order q:

where {\displaystyle \mu }\mu is the mean of the series, the {\displaystyle \theta _{1},...,\theta
_{q}}{\displaystyle \theta _{1},...,\theta _{q}} are the parameters of the model, and the {\
displaystyle \varepsilon _{t},\varepsilon _{t-1},...,\varepsilon _{t-q}}{\displaystyle \varepsilon
_{t},\varepsilon _{t-1},...,\varepsilon _{t-q}} are white noise error terms. The value of q is
called the order of the MA model. This can be equivalently written in terms of the backshift
operator B as.

Thus, a moving-average model is conceptually a linear regression of the current value of the
series against current and previous (observed) white noise error terms or random shocks. The
random shocks at each point are assumed to be mutually independent and to come from the same
distribution, typically a normal distribution, with location at zero and constant scale. Fitting the
MA estimates is more complicated than it is in autoregressive models (AR models), because the
lagged error terms are not observable. This means that iterative non-linear fitting procedures
need to be used in place of linear least squares.
The autocorrelation function (ACF) of an MA (q) process is zero at lag q + 1 and greater.
Therefore, we determine the appropriate maximum lag for the estimation by examining the
sample autocorrelation function to see where it becomes insignificantly different from zero for
all lags beyond a certain lag, which is designated as the maximum lag q.

Sometimes the ACF and partial autocorrelation function (PACF) will suggest that an MA model
would be a better model choice and sometimes both AR and MA terms should be used in the
same model.ARMA models are most commonly estimated using maximum likelihood estimation
(MLE). One consequence of this is that, given some time series and some specified order (p, q),
the estimates obtained from the estimated ARMA (p, q) model will vary depending on the type
of MLE estimation used. As is the case in many situations where one is trying to estimate a time-
series process, model selection is important.

For ARMA models, model selection meaning choosing the number of AR and MA parameters,
the p and q for which a coefficient will be estimated.

In practice, it is common to estimate several different potential models, then use some criterion
to determine which model best fits the time-series.

ARMA models can only be estimated for univariate time series. If you are interested in
estimating a time series process using multiple time series on the right hand side of your model,
consider using a vector AR (VAR) model or a VARMA model.

Before estimating an ARMA model, it is standard practice to try to determine whether or not the
time series appears to be stationary.

In the next chapter we will see the demonstration of its implementation’s using python.
CHAPTER 4

ARMA MODEL

PYTHON
Python is a popular programming language. It was created by Guido van Rossum, and released
in 1991. Python is dynamically-typed and garbage-collected. It supports multiple programming
paradigms, including structured (particularly procedural), object-oriented and functional
programming. It is often described as a "batteries included" language due to its comprehensive
standard library

It is used for: web development (server-side), software development, mathematics, system

scripting.

JULIA:
ARMA p and q models in Julia can be estimated using the StateSpaceModels.jl package, which
also allows for the estimation of a variety of time series models that have linear state-space
representations.
Begin by importing and loading necessary packages into your work environment.

sss
References:

1. Calder, Matthew, and Richard A. Davis. "Introduction to whittle (1953) the analysis of
multiple stationary time series." Breakthroughs in statistics. Springer, New York, NY,
1997. 141-169.

2.