Statistical Machine Learning
Instructor: Prof. Pier Francesco Palamara
University of Oxford, Big Data Institute EPSRC CDT in Health Data Science, Michaelmas 2020
General Information
Course Team  Pier Palamara (module leader, lecturer), Sharib Ali (practical leader) 



Lectures  Monday Oct 19 to Wednesday Oct 21, 9am12pm. 
Practicals  Details on Canvas. 
Practicals
Please check material on Canvas.Syllabus
Recommended prerequisites:
Equivalent of the following Oxford undergraduate courses: Part A A9 Statistics and A8 Probability. SB2a Foundations of Statistical Inference useful by not essential.
Aims and Objectives:
Machine learning studies methods that can automatically detect patterns in data, and then use these patterns to predict future data or other outcomes of interest. It is widely used across many scientific and engineering disciplines. This course covers statistical fundamentals of machine learning, with a focus on supervised learning and empirical risk minimisation. Both generative and discriminative learning frameworks are discussed and a variety of widely used classification algorithms are overviewed.
Synopsis
Visualisation and dimensionality reduction: principal components analysis, biplots and singular value decomposition. Multidimensional scaling. Kmeans clustering. Introduction to supervised learning. Evaluating learning methods with training/test sets. Bias/variance tradeoff, generalisation and overfitting. Crossvalidation. Regularisation. Performance measures, ROC curves. Knearest neighbours as an example classifier. Linear models for classification. Discriminant analysis. Logistic regression. Generative vs Discriminative learning. Naive Bayes models. Decision trees, bagging, random forests, boosting.
Useful reading
C. Bishop, Pattern Recognition and Machine Learning, Springer, 2007.
T. Hastie, R. Tibshirani, J Friedman, Elements of Statistical Learning, Springer, 2009.
K. Murphy, Machine Learning: a Probabilistic Perspective, MIT Press, 2012.
Further Reading
B. D. Ripley, Pattern Recognition and Neural Networks, Cambridge University Press, 1996.
G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning, Springer, 2013.
Slides
Please email me to report any typos or errors.
1  Introduction, unsupervised learning: exploratory data analysis  Slides » 

2  Principal component analysis  Slides » 
3  SVD, Biplots, multidimensional scaling, Isomap  Slides » 
4  Clustering (Kmeans, hierarchical)  Slides » 
5  Supervised learning: empirical risk minimization, regression (linear, polynomial), overfitting  Slides » 
6  Generalization, crossvalidation, biasvariance tradeoff  Slides » 
7  Regularization (demo, source, data), gradient descent, classification: the Bayes classifier  Slides » 
8  Classification: linear and quadratic discriminant analysis (LDA, FDA)  Slides » 
9  QDA, logistic regression  Slides » 
10  Logistic regression (cont.), generative vs. discriminative learning, naïve Bayes  Slides » 
11  Naïve Bayes (cont.), Knearest neighbors, evaluating performance, ROC curves  Slides » 
12  Decision trees  Slides » 
13  Bagging, random forests  Slides » 
14  Boosting  Slides » 
Problem Sheets
This is a set of problems+solutions about the material covered in the lectures above.
1  Sheet »  Solutions » 

2  Sheet »  Solutions » 
3  Sheet »  Solutions » 
4  Sheet »  Solutions » 
Resources for R
 Installation
 RStudio
 Part A Statistical Programming at Oxford
 DataCamp tutorial
 Coursera R programming course
 Intro to tidyverse (advanced)
Background Review Aids
 Matrix and Gaussian identities  short useful reference for machine learning.
 Linear Algebra Review and Reference  useful selection for machine learning.
 Video reviews on Linear Algebra by Zico Kolter.
 Video reviews on Multivariate Calculus and SVD by Aaditya Ramdas.
 The Matrix Cookbook  extensive reference.