Part B Foundations of Statistical Inference (BS2a) MT 2013


Lecturer

Dr Jonathan Marchini  marchini@stats.ox.ac.uk

Course aims 

Understanding how data can be interpreted in the context of a statistical model. Working knowledge and understanding of key-elements of model-based statistical inference, including awareness of similarities, relationships and differences between Bayesian and frequentist approaches.

Synopsis

Exponential families: Curved and linear exponential families; canonical parametrization; likelihood equations.
Sufficiency: Factorization theorem; sufficiency in exponential families.

Frequentist estimation: unbiasedness; method of moments; the Cramer-Rao information inequality;

Rao-Blackwell theorem, Lehmann-Scheffe Theorem and Rao-Blackwellization.
Statement of complete sufficiency for Exponential families.

The Bayesian paradigm: likelihood principal; subjective probability; prior to posterior analysis; asymptotic normality;

conjugacy; examples from exponential families. Choice of prior distribution: proper and improper priors;
Jeffreys and maximum entropy priors. Hierarchical Bayes models.


Computational techniques: Markov chain Monte Carlo methods; The Metropolis-
Hastings algorithm. Gibbs Sampling. Variational Bayesian methods. The EM
algorithm. Approximations to marginal likelihood : Laplace approximation and
BIC.


Decision theory: risk function; Minimax rules, Bayes rules. Point estimators and admissability of Bayes rules.

The James-Stein estimator, shrinkage estimators and Empirical Bayes. Hypothesis testing as decision problem.

Pre-course reading

A brief revision of the material covered in Part A Statistics course would be a good idea.
Course notes for Part A Statistics can be found here.

Lectures

Mondays 2.00pm and Thursdays 10.00am weeks 1-8 in the Statistics Department, 1 South Parks Road.

Lecture slides

Copies of lecture slides will usually appear here before each lecture.
I recommend that you bring copies of the slides to each lecture.
The 4-up slides would be best for this purpose.
For some lectures there is R code that implements examples illustrated in the lectures.

Lecture
Date
Topic
1up slides
4up slides
R code
1
14/10/13
Exponential families
lecture1.pdf
lecture1_4up.pdf

2
16/10/13
Sufficiency, Factorization Theorem, Minimal sufficiency
lecture2.pdf lecture2_4up.pdf
3
21/10/13
Estimators, Minimum Variance Unbiased Estimators (MVUE)
The Cramér-Rao Lower Bound.
lecture3.pdf lecture3_4up.pdf
4
24/10/13
Consequences of the Cramér-Rao Lower Bound.
Searching for a MVUE. Rao-Blackwell Theorem, Lehmann-Scheffé Theorem.
lecture4.pdf lecture4_4up.pdf
5
28/10/13
The method of moments
lecture5.pdf lecture5_4up.pdf
6
31/10/13
Bayesian Inference.
lecture6.pdf lecture6_4up.pdf
7
04/10/13
Prior distributions. Hierarchical models. Predictive distributions.
lecture7.pdf lecture7_4up.pdf
8
07/10/13
Bayesian hypothesis testing.
lecture8.pdf lecture8_4up.pdf
9
11/11/13
Decision theory. Loss functions. Risk functions. Admissible and inadmissible rules. Minimax. Bayes Risk and Bayes rules. lecture9.pdf lecture9_4up.pdf
10
14/11/13
Finding minimax rules. Hypothesis testing with loss functions. lecture10.pdf lecture10_4up.pdf
11
18/11/13
Stein's paradox and the James-Stein estimator. Empirical Bayes
lecture11.pdf lecture11_4up.pdf
12
21/11/13
Markov Chain Monte Carlo. The Metropolis algorithm.
lecture12.pdf lecture12_4up.pdf lecture12.R
13
25/11/13
MCMC convergence. The Metropolis-Hastings algorithm. The Gibbs sampler.
lecture13.pdf lecture13_4up.pdf lecture13.R
14
27/11/13
Variational Bayes
lecture14.pdf lecture14_4up.pdf
15
01/12/13
The EM algorithm
lecture15.pdf lecture15_4up.pdf
16
05/11/13
Bayesian analysis of contingency tables. Bayesian linear regression.
lecture16.pdf lecture16_4up.pdf

PDF of all 16 lectures in 4up format :
bs2a_4up.pdf

Classes and problem sheets

There will be classes in weeks 3-8 on Fridays.

Work should be handed in to the appropriate BS2a tray in SPR1 by 12noon on Wednesdays.

Time
Location
Tutor
Assistant
12-1
SPR1 seminar room
Charlotte Greenan
Zhu Li
4-5
SPR1 seminar room Jonathan Marchini
Andreas Anastasiou
4-5
SPR2 seminar room Charlotte Greenan Zhu Li
5-6
SPR1 seminar room Charlotte Greenan Andreas Anastasiou

Class
Sheet
Week 3
ex1.pdf
Week 4
ex2.pdf
Week 5
ex3.pdf
Week 6
ex4.pdf
Week 7
ex5.pdf
Week 8
ex6.pdf

Reading

  1. P. H. Garthwaite, I. T. Jolliffe and Byron Jones, Statistical Inference, Second ed. Oxford University Press, 2002
  2. G.A.Young and R.L. Smith,  Essentials of Statistical Inference, Cambridge University  Press, 2005. 
  3. T. Leonard and J.S.J. Hsu, Bayesian Methods, Cambridge University Press, 2005.

Further reading

  1. D. R. Cox, Principles of Statistical Inference, Cambridge University Press, 2006
  2. H. Liero and S Zwanzig, Introduction to the Theory of Statistical Inference, CRC Press,  2012
  3. D. Barber, Bayes Reasoning and Machine Learning, Cambridge University Press,
    2012