Part B Foundations of Statistical Inference (SB2a) MT 2016


Lecturer

Dr Julien Berestycki   julien.berestycki@stats.ox.ac.uk

Course aims 

Understanding how data can be interpreted in the context of a statistical model. Working knowledge and understanding of key-elements of model-based statistical inference, including awareness of similarities, relationships and differences between Bayesian and frequentist approaches.

Revisions and Consultations  

Revision classes, Wednesday, weeks 2 (2013 exam paper) and 5, 10am, LG.01.
Consultation sessions, Wednesday, weeks 2 and 5, 1 pm-2pm, LG.04.



Synopsis

Exponential families: Curved and linear exponential families; canonical parametrization; likelihood equations.
Sufficiency: Factorization theorem; sufficiency in exponential families.

Frequentist estimation: unbiasedness; method of moments; the Cramer-Rao information inequality;

Rao-Blackwell theorem, Lehmann-Scheffe Theorem and Rao-Blackwellization.
Statement of complete sufficiency for Exponential families.

The Bayesian paradigm: likelihood principal; subjective probability; prior to posterior analysis; asymptotic normality;

conjugacy; examples from exponential families. Choice of prior distribution: proper and improper priors;
Jeffreys and maximum entropy priors. Hierarchical Bayes models.


Computational techniques: Variational Bayesian methods. The EM
algorithm. Approximations to marginal likelihood : Laplace approximation and
BIC.


Decision theory: risk function; Minimax rules, Bayes rules, randomised rules. Point estimators and admissibility of Bayes rules.

The James-Stein estimator, shrinkage estimators and Empirical Bayes. Hypothesis testing as decision problem.

Pre-course reading

A brief revision of the material covered in Part A Statistics course would be a good idea.
Course notes for Part A Statistics can be found here.

Lectures

Mondays 3pm and Tuesdays 11am weeks 1-8 in the Statistics Department.

Lecture slides

Copies of lecture slides will usually appear here before each lecture.
I recommend that you bring copies of the slides to each lecture.
The 4-up slides would be best for this purpose.
For some lectures there is R code that implements examples illustrated in the lectures.

Lecture
Date
Topic
1up slides
4p slides
R code
1
10/10/16
Exponential families
lecture1.pdf
lecture1_4p.pdf

2
11/10/16
Sufficiency, Factorization Theorem, Minimal sufficiency
lecture2.pdf
lecture2_4p.pdf

3
17/10/16
Estimators, Minimum Variance Unbiased Estimators (MVUE)
The Cramér-Rao Lower Bound.
lecture3.pdf lecture3_4p.pdf
4
18/10/16
Consequences of the Cramér-Rao Lower Bound.
Searching for a MVUE. Rao-Blackwell Theorem.
lecture4.pdf lecture4_4p.pdf
5
24/10/16
Lehmann-Scheffé Theorem and completeness. The method of moments
lecture5.pdf lecture5_4p.pdf
6
25/10/16
Bayesian Inference.
lecture6.pdf lecture6_4p.pdf
7
31/10/16
Prior distributions. Predictive distributions. Normal approximations.
lecture7.pdf lecture7_4p.pdf
8
01/11/16
Hierarchical models
lecture8.pdf lecture8_4p.pdf
9
7/11/16
Bayesian hypothesis testing. lecture9.pdf lecture9_4p.pdf
10
8/11/16
Decision theory. Loss functions. Risk functions. Admissible and inadmissible rules. Minimax. Bayes Risk and Bayes rules. Randomised rules. lecture10.pdf lecture10_4p.pdf
11
14/11/16
Finding minimax rules. Hypothesis testing with loss functions.
lecture11.pdf lecture11_4p.pdf
12
15/11/16
Stein's paradox and the James-Stein estimator.
lecture12.pdf lecture12_4p.pdf
13
21/11/16
Empirical Bayes.
lecture13.pdf lecture13_4p.pdf
14
22/11/16
Variational Bayes.
lecture14.pdf lecture14_4p.pdf
15
28/11/16
The EM algorithm (draft version)
lecture15.pdf lecture15_4p.pdf
16
29/11/16
Bayesian analysis of contingency tables. Bayesian linear regression.
lecture16.pdf lecture16_4p.pdf Bayes_table.R


Classes and problem sheets

For UG students: There will be classes in weeks 3,5,7 and 8 on Wed.(10am and 11:30 am, tutor J. berestycki) and in weeks 3,4,7 and 8 on Thur. (9:30am and 12 am, tutor S. Filippi)

Doodle link to register for classes

For UG students only: Work should be handed in to the appropriate BS2 tray (by noon the Monday prior to the class).

Public
Time
Location
Tutor
Assistant
Weeks
U.G.
Wed. 10-11:30 am
LG.04
Julien Beresycki
TBA
3,5,7,8
U.G.
Wed. 11:30am-1pm
LG.04
Julien Beresycki
TBA
3,5,7,8
U.G.
Thur. 9:30-11:00 am
LG.05
Sarah Filippi
TBA
3,4,7,8
U.G.
Thur. 12am-1:30pm
LG.05
Sarah Filippi
TBA
3,4,7,8
msc
Thur. 9am-10am
LG.01
Julien Berestycki
NA
3,5,7,8


Class
Sheet
Week 3
Sheet1.pdf
Week 4
Sheet2.pdf Version for week 4
Week 5
Sheet2.pdf Version for week 5
Week 7
Sheet3.pdf
Week 8
Sheet4.pdf

Reading

  1. P. H. Garthwaite, I. T. Jolliffe and Byron Jones, Statistical Inference, Second ed. Oxford University Press, 2002
  2. G.A.Young and R.L. Smith,  Essentials of Statistical Inference, Cambridge University  Press, 2005. 
  3. T. Leonard and J.S.J. Hsu, Bayesian Methods, Cambridge University Press, 2005.

Further reading

  1. D. R. Cox, Principles of Statistical Inference, Cambridge University Press, 2006
  2. H. Liero and S Zwanzig, Introduction to the Theory of Statistical Inference, CRC Press,  2012
  3. D. Barber, Bayes Reasoning and Machine Learning, Cambridge University Press,
    2012