Probability (and Statistics) --- Lecturer: Kuen-Tsair Lay

Course outline (brief)

 Classical Probability and Counting
 Axioms of Probability and Basic Theorems
 Conditional Probability and Independence
 Discrete Random Variables
 Special Discrete Distributions
 Continuous Random Variables
1
 Special Continuous Distributions
 Jointly distributed random variables
 Conditioning by Random Variables
 Sums of Independent Random Variables and Limit
Theorems

Books
 Textbook: H. Tijms, “Probability: A Lively Introduction,”
Cambridge Univ. Press, Cambridge, UK, 2018
2
 (distributed by 滄海書局
(https://www.tsanghai.com.tw/) in Taiwan)
 Reference books:
 S. Ghahramani, “Fundamentals of Probability, with
rd
Stochastic Processes,” 3 ed., Pearson Prentice Hall,
NJ 07458, USA, 2005
 R. E. Walpole and R. H. Myers, “Probability and
th
Statistics for Engineers and Scientists,” 5 ed.,
Macmillan Publishing Co., NY 10022, USA, 1993
3
Grading
 Quizzes: 20%
 Midterm exam: 40%
 Final exam: 40%
All exams are open-book tests.

Detailed Course outline

 Classical Probability and Counting

4
 Brief history about probability (Cardano, Fermat,
Pascal, Huygens, Bernoulli, Laplace, etal.)
 Axiomization of probability by Kolmogorov (1933)
 Classical (or, relative-frequency; or, proportion)
view/interpretation of probability
 Combinatorial counting (fundamental principle of
counting, permutations, combinations, etc.)
 A poker game simulation App (@StatCrunch)
 Binomial/multinomial expansion
5
  Examples and problems
 The relative-frequency interpretation can be
linked-to/derived-from the axiomatic probability
theory via the law of large numbers.
 Axioms of Probability and Basic Theorems
 Axiomization was established by Kolmogorov in 1933.
 Sample space and sample points
 Events (in essence, subsets)
 Basic terminologies and operations on sets
6
  The three axioms
 Increasing/decreasing sequences of events
 Theorem of continuity of probability
 Some basic theorems/rules (complement,
inclusion-exclusion, etc.)
 Examples and problems
 Conditional Probability and Independence
 Concept/definition of conditional probability

7
  Multiplication rule and chain rule for joint events via
conditional probabilities
 Independent events
 Law of totality
 Bayes’ rule – basic/odds from
 Examples and problems
 Discrete Random Variables
 Concept of a random variable (r.v.)
 Discrete vs. continuous random variables
8
 Probability mass function (pmf)
 Distribution
 Mean
 Expected values and the substitution rule
 Moments
 Variance
 Mean of sum of r.v.’s
 Joint vs. marginal pmf’s
 Independence between r.v.’s
9
  Mean of product of independent r.v.’s
 Variance of sum of independent r.v.’s
 Special Discrete Distributions
 Bernoulli distribution
 Binomial distribution
 Geometric distribution
 Poisson distribution
 Hypergeometric distribution
 Negative binomial distribution
10
  Examples and problems
 Continuous Random Variables
 Probability density function (pdf)
 Cumulative distribution function (cdf)
 Pdf of a function of a r.v. – method of distribution
and method of transformation
 Mean
 Expected values and the substitution rule
 Variance
11
 Special Continuous Distributions
 Uniform distribution
 Exponential distribution
 Gamma distribution
 Weibull distribution
 Beta distribution
 Normal/Gaussian distribution
 Chi-square distribution
 Student-t distribution
12
  F distribution
 Rayleigh distribution
 Rician distribution
 Inverse-transformation method and simulation
 Examples and problems
 Jointly distributed random variables
 Focus is on bi-variate case
 Generalization to multi-variate cases is
straight-forward (usually).
13
 Joint probability mass function – discrete r.v.’s
 Marginal probability mass function
 Joint probability density function – continuous r.v.’s
 Joint (cumulative) distribution function
 Marginal probability density function
 Expected value of sum of r.v.’s
 Independence of r.v.’s
 Conditional distribution (to be elaborated later)
 Variance of sum of independent r.v.’s
14
 Pdf/pmf of sum of independent r.v.’s – convolution
of individual pdf/pmf
 Transformation of two r.v.’s
 Covariance and correlation coefficient
 Uncorrelatedness vs. independence
 Linear predictor and regression
 Bi-variate Gaussian distribution
 Multi- variate Gaussian distribution
 Examples and problems
15
 Conditioning by Random Variables
 Conditional probability mass function
 Conditional probability density function
 Law of conditional probability (in essence, a special
case of law of totality)
 Law of conditional expectation/mean
 Minimum mean squared-error estimator and
conditional mean

16
  Bayesian inference – estimation of parameters in a
distribution
 A-priori probability, likelihood, and a-posteriori
probability
 Examples and problems
 Sums of Independent Random Variables and Limit
Theorems
 Generating function of (nonnegative) integer-valued
r.v. (in essence, z-transform for discrete-time signal)
17
 Convolution rule in terms of generating functions for
independent r.v.’s
 Inversion of generating functions – contour
integration of complex-valued functions
 Inversion of generating functions – table look-up
 Moment-generating function (mgf)
 Relating moment-generating function to Fourier
transform
 Moment-generating functions of some distributions
18
 Convolution rule in terms of moment-generating
functions for independent r.v.’s
 Sum of independent r.v.’s (mgf and pdf)
 Sum of independent binomial r.v.’s
 Sum of independent Poisson r.v.’s
 Sum of independent Gaussian r.v.’s
 Sum of independent gamma r.v.’s
 Sum of independent squared standard-Gaussian r.v.’s
– Chi-square
19
 Markov inequality
 Chebyshev inequality
 Law of large numbers – weak version
 Law of large numbers – strong version
 Relative-frequency interpretation of probability via
the law of large numbers
 Central limit theorem
 Enhanced central limit theorem (regularity condition
is weakened)
20