Statistical Machine Learning
Prof. Pier Palamara, University of Oxford, Hilary term 2020
Course offered to Part B students (SB2b) and MSc students (SM4)
New: Revisions (Trinity Term)
You can access past exam questions via OXAM, which is on WebLearn [solutions here]. Note that the syllabus of this course is different from previous editions (e.g. it used to be a Part C course). Past exam questions for revision include:
 Part B 2017 Q1, Q2 (excluding c), Q3
 Part C 2016 Q1 (ab), Q2 (ab)
 Part C 2015 Q1, Q2 (ac), Q3 (b)
 MSc 2017 Q7, Q8 (excluding bc)
 MSc 2014 Q7
 MSc 2012 Q6, Q7
 MSc and Part B 2018
 MSc and Part B 2019
Revision Class 1  Week 2, recorded by Brian Zhang. Plan: 2015 Part C Q1 and Q2 (ac), Q3 (b) and selected parts of 2017 MSc Q7. 

Revision Class 2  Week 3, recorded by Juba Nait Saada. Plan: 2018 Part B exam.. 
Consultation Session 1  Week 4, Wednesday 20th May, 4pm5pm with Pier Palamara. To make the consultation sessions more efficient, please send your questions by email beforehand. (see email in General Info). 
Consultation Session 2  Week 5, Wednesday 27th May, 10am11am with Tom Rainforth. To make the consultation sessions more efficient, please send your questions by email beforehand. (see email in General Info). 
General Information
Course Team  Pier Palamara (13 lectures + tutor), Tom Rainforth (3 lectures + tutor) Brian Zhang (tutor), Juba Nait Saada (tutor), David Rindt (TA), Tom Hadfield (TA), Zhongyi Hu (TA) 

Lecturers:
Tutors: TAs: 

Lectures  HT Weeks 18, Statistics room LG.01, Tue 5pm6pm, Thu 4pm5pm. Note: Second lecture moved to Wed 9am10am. 
Classes  HT weeks 3/4, 5/6, 7/8, TT Week 1 (details below) 
Practicals  MSc only: HT Week 4, Friday 11am1pm (unassessed) and HT Week 8, Friday 11am1pm (assessed, groups of 4) 
Tutorials
Please contact to change group or any other matter related to class allocation.Group  Time (location)  Class Tutor / TA 

SB2B group 1  Monday 11.15am12.45pm, weeks 3, 5, 7 in LG.03, TT1 in LG.05.  Pier Palamara / Zhongyi Hu 
SB2B group 2  Tuesday 11am12.30pm, weeks 3, 5, 7 in LG.03, TT1 in LG.05  Brian Zhang / Tom Hadfield 
SB2B group 3  Tuesday 2.30pm4pm, weeks 3, 5, 7 in LG.03, TT1 in LG.05  Brian Zhang / Tom Hadfield 
SB2B group 4  Wednesday 9.30am11am, weeks 3, 5, 7, TT1 in LG.05  Juba Nait Saada / Zhongyi Hu 
SB2B group 5  Monday 10am11.30am, weeks 4, 6, 8 in LG.04, TT1 in LG.02  Tom Rainforth / David Rindt 
SB2B group 6  Monday 11.30am1pm, weeks 4, 6, 8 in LG.04, TT1 in LG.02  Tom Rainforth / David Rindt 
SM4 (single group)  Monday 3pm4pm, weeks 3, 5, 7, TT1 in LG.01  Pier Palamara 
Syllabus
Recommended prerequisites:
Part A A9 Statistics and A8 Probability. SB2a Foundations of Statistical Inference useful by not essential.
Aims and Objectives:
Machine learning studies methods that can automatically detect patterns in data, and then use these patterns to predict future data or other outcomes of interest. It is widely used across many scientific and engineering disciplines. This course covers statistical fundamentals of machine learning, with a focus on supervised learning and empirical risk minimisation. Both generative and discriminative learning frameworks are discussed and a variety of widely used classification algorithms are overviewed.
Synopsis
Visualisation and dimensionality reduction: principal components analysis, biplots and singular value decomposition. Multidimensional scaling. Kmeans clustering. Introduction to supervised learning. Evaluating learning methods with training/test sets. Bias/variance tradeoff, generalisation and overfitting. Crossvalidation. Regularisation. Performance measures, ROC curves. Knearest neighbours as an example classifier. Linear models for classification. Discriminant analysis. Logistic regression. Generative vs Discriminative learning. Naive Bayes models. Decision trees, bagging, random forests, boosting. Neural networks and deep learning.
Reading
C. Bishop, Pattern Recognition and Machine Learning, Springer, 2007.
T. Hastie, R. Tibshirani, J Friedman, Elements of Statistical Learning, Springer, 2009.
K. Murphy, Machine Learning: a Probabilistic Perspective, MIT Press, 2012.
Further Reading
B. D. Ripley, Pattern Recognition and Neural Networks, Cambridge University Press, 1996.
G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning, Springer, 2013.
Slides
Slides will be made available as the course progresses and periodically updated. Please check back for updates.
You can use the material from last year (here) to get a sense of future lectures. The schedule below may change.
Please email us to report any typos or errors.
1  (PP) Introduction, unsupervised learning: exploratory data analysis  Slides » 

2  (PP) Principal component analysis  Slides » 
3  (PP) SVD, Biplots, multidimensional scaling, Isomap  Slides » 
4  (PP) Clustering (Kmeans, hierarchical)  Slides » 
5  (PP) Supervised learning: empirical risk minimization, regression (linear, polynomial), overfitting  Slides » 
6  (PP) Generalization, crossvalidation, biasvariance tradeoff  Slides » 
7  (PP) Regularization, (demo, source, data) gradient descent, classification: the Bayes classifier  Slides » 
8  (PP) Classification: linear and quadratic discriminant analysis (LDA, FDA)  Slides » 
9  (PP) QDA, logistic regression  Slides » 
10  (PP) Logistic regression (cont.), generative vs. discriminative learning, naïve Bayes  Slides » 
11  (PP) Naïve Bayes (cont.), Knearest neighbors, evaluating performance, ROC curves  Slides » 
12  (PP) Neural networks  Slides » 
13  (PP) Boosting  Slides » 
14  (TR) Decision Trees  Slides » 
15  (TR) Bagging, random forests  Slides » 
16  (TR) Deep learning  Slides » 
Problem Sheets
For undergraduate students: hand in solutions in the pigeon holes labeled with your class time.
Class allocation details are on Minerva (accessible from Oxford network).
1  due at noon, January 31st  Sheet »  Solutions » 

2  due at noon, February 14th  Sheet »  Solutions » 
3  due at noon, February 28st  Sheet »  Solutions » 
4  due at noon, April 24th  Sheet »  Solutions » 
Resources for R
 Installation
 RStudio
 Part A Statistical Programming at Oxford
 DataCamp tutorial
 Coursera R programming course
 Intro to tidyverse (advanced)
Background Review Aids
 Matrix and Gaussian identities  short useful reference for machine learning.
 Linear Algebra Review and Reference  useful selection for machine learning.
 Video reviews on Linear Algebra by Zico Kolter.
 Video reviews on Multivariate Calculus and SVD by Aaditya Ramdas.
 The Matrix Cookbook  extensive reference.