Harvard University, Fall 2013

Prof. Ryan Adams (OH: Mon 2:30-3:30pm in MD 233)TF: Eyal Dechter (OH: Thu 1pm in MD 1st Floor Lounge; Section: Thu 2:30-3:30pm in MD 319)

TF: Scott Linderman (OH: Thu 10am in MD 2nd Floor Lounge; Section: Thu 9-10am in MD 221)

TF: Dougal Maclaurin (OH: Mon 10am in MD 334; Section: Fri 10-11am in MD 223)

Time: Monday and Wednesday, 1-2:30pm

Location: Maxwell-Dworkin G115

Contact: (course number) + (hyphen) + (the word "f13") + (hyphen) + (the word "staff") + "seas.harvard.edu"

- 2 December 2013: The final project poster session will be on the ground and first floors of Maxwell-Dworkin on Thursday 5 December from 2-4pm. Come a bit early to set things up.
- 3 November 2013: Assignment 5 is now available.
- 25 October 2013: Assignment 4 is now available.
- 18 October 2013: A practice midterm is available.
- 11 October 2013: A list of midterm study topics is now available.
- 6 October 2013: Assignment 3 is now available. The due date is extended to October 20.
- 21 September 2013: Assignment 2 is now available.
- 13 September 2013: Section times and places, as well as office hours, are now available.
- 10 September 2013: Please identify your section preferences here.
- 9 September 2013: Due to the size of the class, we are moving to a larger room: G115.
- 8 September 2013: If you filled out the survey, you should have now received notification of your status. If you have been assigned a place but do not intend to take the course, please let the staff know ASAP so that another student can take the course.
- 6 September 2013: Assignment 1 is now available.
- 5 September 2013: It has become apparent that cross-registered students will not be able to access the online quizzes until after study card day. As such, there will not be a quiz for the second lecture.
- 3 September 2013: Please fill out the survey.
- 31 August 2013: The syllabus is now available.
- 28 August 2013: Sign up on the Piazza discussion site.

Wed 4 Sep 2013

Mon 9 Sep 2013

Lecture 1: Introduction to Inference and Learning

- [required] Book: Murphy -- Chapter 1 --
*Introduction* - [optional] Book: Bishop -- Chapter 1 --
*Introduction* - [optional] Video: Christopher Bishop -- Embracing Uncertainty: The New Machine Intelligence
- [optional] Video: Sam Roweis -- Machine Learning, Probability and Graphical Models, Part 1
- [optional] Video: Iain Murray -- Introduction to Machine Learning, Part 1
- [optional] Video: Neil Lawrence -- What is Machine Learning?

Mon 9 Sep 2013

Lecture 2: Simple Discrete Models [ discrete_coins.m | plot_beta.m | beta_coins.m ]

- Assignment 1 Out
- [required] Book: Murphy -- Chapter 2 --
*Probability* - [required] Book: Murphy -- Chapter 3 --
*Generative Models for Discrete Data* - [optional] Book: Bishop -- Chapter 2, Sections 2.1-2.2 --
*Probability Distributions* - [optional] Book: MacKay -- Chapter 2 --
*Probability, Entropy, and Inference* - [optional] Book: MacKay -- Chapter 3 --
*More About Inference* - [optional] Book: Mackay -- Chapter 23 --
*Useful Probability Distributions* - [optional] Video: Iain Murray -- Introduction to Machine Learning, Part 2
- [optional] Metacademy: Bayesian Parameter Estimation
- [optional] Metacademy: Dirichlet Distribution

Lecture 3: Simple Gaussian Models [ plot_bigauss1.m | plot_bigauss2.m ]

- [required] Book: Murphy -- Chapter 4 --
*Gaussian Models* - [optional] Book: Bishop -- Chapter 2, Section 2.3 --
*Probability Distributions* - [optional] Note: David J.C. MacKay --
*The Humble Gaussian Distribution*. - [optional] Note: Sam Roweis --
*Gaussian Identities*. - [optional] Metacademy: Multivariate Gaussian Distribution

Fri 13 Sep 2013

Section 1: Math Review [ notes ]

- [optional] Note: Sam Roweis --
*Matrix Identities*. - [optional] Book: Mackay -- Appendix A --
*Notation* - [optional] Metacademy: Probability
- [optional] Metacademy: Lnear Algebra
- [optional] Metacademy: Multivariate Calculus

Mon 16 Sep 2013

Lecture 4: Bayesian Statistics [ quiz ]

- [required] Book: Murphy -- Chapter 5 --
*Bayesian Statistics* - [optional] Book: Murphy -- Chapter 6 --
*Frequentist Statistics* - [optional] Video: Iain Murray -- Introduction to Machine Learning, Part 3
- [optional] Video: Michael Jordan -- Bayesian or Frequentist: Which Are You?
- [optional] Video: Christopher Bishop -- Introduction to Bayesian Inference
- [optional] Metacademy: Bayesian Decision Theory

Wed 18 Sep 2013

Lecture 5: Linear Regression (Guest Lecturer: Matt Johnson) [ demos ]

- Assignment 1 Help Session 5-7pm, Maxwell-Dworkin Second Floor Lounge
- [required] Book: Murphy -- Chapter 7 --
*Linear Regression* - [optional] Book: Bishop -- Chapter 3 --
*Linear Models for Regression* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 2 --
*Overview of Supervised Learning* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 3 --
*Linear Methods for Regression* - [optional] Metacademy: Linear Regression

Fri 20 Sep 2013

Section 2: Practical Optimization [ notes ]

- Assignment 1 Due
- Assignment 2 Out
- [optional] Metacademy: Optimization

Mon 23 Sep 2013

Lecture 6: Linear Classifiers [ quiz ]

- [required] Book: Murphy -- Chapter 8 --
*Logistic Regression* - [optional] Book: Bishop -- Chapter 4 --
*Linear Models for Classification* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 4 --
*Linear Methods for Classification* - [optional] Metacademy: Binary Linear Classifiers

Wed 25 Sep 2013

Lecture 7: Generalized Linear Models [ quiz ]

- [required] Book: Murphy -- Chapter 9, Sections 9.1-9.4 --
*Generalized Linear Models and the Exponential Family* - [optional] Book: Murphy -- Chapter 9, Sections 9.5-9.7 --
*Generalized Linear Models and the Exponential Family* - [optional] Book: Koller and Friedman -- Chapter 8 --
*The Exponential Family* - [optional] Paper: Kevin P. Murphy --
*Conjugate Bayesian Analysis of the Gaussian Distribution*, informal note, 2007. - [optional] Video: Alex Smola -- Exponential Families, Part I
- [optional] Metacademy: Generalized Linear Models
- [optional] Metacademy: Exponential Family

Section 3: Undirected Graphical Models and Factor Graphs [ notes ]

- Return Assignment 1
- [required] Book: Murphy -- Chapter 19, Sections 19.1-19.4 --
*Undirected Graphical Models (Markov Random Fields)* - [optional] Book: Bishop -- Chapter 8, Sections 8.1-8.3 --
*Graphical Models* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 17 --
*Undirected Graphical Models* - [optional] Book: Koller and Friedman -- Chapter 4 --
*Undirected Graphical Models* - [optional] Metacademy: Markov Random Fields
- [optional] Metacademy: Factor Graphs

Mon 30 Sep 2013

Lecture 8: Directed Graphical Models [ quiz ]

- [required] Book: Murphy -- Chapter 10, Sections 10.1-10.5 --
*Directed Graphical Models (Bayes Nets)* - [optional] Book: Bishop -- Chapter 8 --
*Graphical Models* - [optional] Book: Koller and Friedman -- Chapter 3 --
*The Bayesian Network Representation* - [optional] Paper: Martin J. Wainwright and Michael I. Jordan.
*Graphical Models, Exponential Families and Variational Inference*. Foundations and Trends in Machine Learning 1(1-2):1-305, 2008. - [optional] Paper: Michael I. Jordan.
*Graphical Models*. Statistical Science 19(1):140-155, 2004. - [optional] Video: Zoubin Ghahramani -- Graphical Models
- [optional] Video: Cedric Archambeau -- Graphical Models
- [optional] Metacademy: Bayesian Networks

Wed 2 Oct 2013

Lecture 9: Mixture Models [ quiz ]

- Assignment 2 Help Session, 5-7pm, Maxwell-Dworkin Second Floor Lounge
- [required] Book: Murphy -- Chapter 11 --
*Mixture Models and the EM Algorithm* - [optional] Book: Bishop -- Chapter 9 --
*Mixture Models and EM* - [optional] Book: Mackay -- Chapter 20 --
*An Example Inference Task: Clustering* - [optional] Book: Mackay -- Chapter 22 --
*Maximum Likelihood and Clustering* - [optional] Metacademy: Expectation-Maximization Algorithm

Fri 4 Oct 2013

Section 4: Factor Analysis and PCA [ notes ]

- Assignment 2 Due
- Assignment 3 Out
- [required] Book: Murphy -- Chapter 12 --
*Latent Linear Models* - [required] Paper: Sam Roweis and Zoubin Ghahramani.
*A Unifying Review of Linear Gaussian Models*. Neural Computation 11(2), 1999. - [optional] Book: Bishop -- Chapter 12, Sections 12.1-12.2 --
*Continuous Latent Variables* - [optional] Book: MacKay -- Chapter 34 --
*Independent Component Analysis and Latent Vriable Modelling* - [optional] Video: Aapo Hyvarinen -- Independent Components Analysis
- [optional] Metacademy: Factor Analysis
- [optional] Metacademy: Principal Component Analysis

Mon 7 Oct 2013

Lecture 10: Sparse Linear Models [ quiz ]

- [required] Book: Murphy -- Chapter 13, Sections 13.1-13.7 --
*Sparse Linear Models* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 3 --
*Linear Methods for Regression* - [optional] Metacademy: LASSO

Lecture 11: Exact Inference [ quiz ]

- [required] Book: Murphy -- Chapter 17, Section 17.1-17.4 --
*Markov and Hidden Markov Models* - [required] Book: Murphy -- Chapter 20, Section 20.1-20.3 --
*Exact Inference for Graphical Models* - [optional] Book: Murphy -- Chapter 17, Section 17.5-17.6 --
*Markov and Hidden Markov Models* - [optional] Book: Bishop -- Chapter 8, Sections 8.4 --
*Graphical Models* - [optional] Book: Bishop -- Chapter 13, Sections 13.1-13.2 --
*Sequential Data* - [optional] Book: Mackay -- Chapter 16 --
*Message Passing* - [optional] Book: Mackay -- Chapter 21 --
*Exact Inference by Complete Enumeration* - [optional] Book: Mackay -- Chapter 24 --
*Exact Marginalization* - [optional] Book: Mackay -- Chapter 26 --
*Exact Marginalization in Graphs* - [optional] Book: Koller and Friedman -- Chapter 9 --
*Exact Inference: Variable Elimination* - [optional] Book: Koller and Friedman -- Chapter 10 --
*Exact Inference: Clique Trees* - [optional] Book: Koller and Friedman -- Chapter 13 --
*MAP Inference* - [optional] Frank R. Kschischang, Brendan J. Frey and Hans-Andrea Loeliger.
*Factor Graphs and the Sum-Product Algorithm*. IEEE Transactions on Information Theory 47(2):498-519, 2001. - [optional] Metacademy: Variable Elimination

Fri 11 Oct 2013

Section 5: The Junction Tree Algorithm [ notes ]

- Last Day for Assignment 1 Regrades
- Return Assignment 2
- [required] Book: Murphy -- Chapter 20, Section 20.4 --
*Exact Inference for Graphical Models* - [optional] Metacademy: Junction Trees

Wed 16 Oct 2013

Lecture 12: Variational Inference (Guest Lecturer: Matt Johnson) [ demo | quiz ]

- Assignment 3 Help Session 5-7pm, Maxwell-Dworkin Second Floor Lounge
- [required] Book: Murphy -- Chapter 21, Section 21.1-21.3, 21.5-21.6 --
*Variational Inference* - [optional] Book: Bishop -- Chapter 10 --
*Approximate Inference* - [optional] Book: MacKay -- Chapter 27 --
*Laplace's Method* - [optional] Book: MacKay -- Chapter 33 --
*Variational Methods* - [optional] Paper: Winn and Bishop -- Variational Message Passing
- [optional] Paper: Thomas Minka -- Divergence Measures and Message Passing
- [optional] Paper: Wainwright, Jaakkola and Willsky -- A New Class of Upper Bounds on the Log Partition Function
- [optional] Video: Tom Minka -- Approximate Inference
- [optional] Video: Martin Wainwright -- Graphical Models, Variational Methods and Message Passing
- [optional] Video: Christopher Bishop -- Graphical Models and Variational Methods
- [optional] Video: David Sontag -- Approximate Inference in Graphical Models using LP relaxations
- [optional] Metacademy: Variational Inference

Fri 18 Oct 2013

Section 6: Loopy Belief Propagation

- Assignment 3 Due
- [required] Book: Murphy -- Chapter 22, Section 22.1-22.2, 22.5-22.6 --
*More Variational Inference* - [optional] Paper: Thomas Minka.
*Divergence Measures and Message Passing*. Microsoft Research MSR-TR-2005-173, 2005. - [optional] Paper: John Winn and Christopher Bishop.
*Variational Message Passing*. Journal of Machine Learning Research 6:661-694, 2005. - [optional] Metacademy: Loopy Belief Propagation

Mon 21 Oct 2013

Lecture 13: Monte Carlo Basics (Guest Lecturer: Finale Doshi-Velez) [ quiz ]

- [required] Book: Murphy -- Chapter 23, Section 23.1-23.4 --
*Monte Carlo Inference* - [optional] Book: Bishop -- Chapter 11, Section 11.1 --
*Sampling Methods* - [optional] Book: MacKay -- Chapter 29 --
*Monte Carlo Methods* - [optional] Book: Devroye -- Chapter 2
- [optional] Metacademy: Monte Carlo Estimation
- [optional] Metacademy: Rejection Sampling
- [optional] Metacademy: Importance Sampling

Wed 23 Oct 2013

Midterm Exam

- Final Project Brainstorming Session 5-7pm, Maxwell-Dworkin Second Floor Lounge

Fri 25 Oct 2013

Section 7: Particle Filtering

- Final Project Proposals Due
- Assignment 4 Out
- Assignment 3 Back
- Last Day for Assignment 2 Regrades
- [required] Book: Murphy -- Chapter 23, Section 23.5-23.6 --
*Monte Carlo Inference* - [optional] Book: Bishop -- Chapter 13, Section 13.3 --
*Sequential Data* - [optional] Paper: Arnaud Doucet and Adams M. Johansen.
*A Tutorial on Particle Filtering and Smoothing: Fifteen Years Later*, 2008. - [optional] Video: Simon Godsill -- Particle Filters
- [optional] Video: Arnaud Doucet and Nando de Freitas -- Sequential Monte Carlo Methods

Mon 28 Oct 2013

Lecture 14: Markov Chain Monte Carlo [ quiz ]

- [required] Book: Murphy -- Chapter 24, Sections 24.1-24.4 --
*Markov Chain Monte Carlo (MCMC) Inference* - [optional] Book: Bishop -- Chapter 11, Section 11.2-11.3 --
*Sampling Methods* - [optional] Book: MacKay -- Chapter 30 --
*Efficient Monte Carlo Methods* - [optional] Paper: Christophe Andrieu, Nando de Freitas, Arnaud Doucet and Michael I. Jordan.
*An Introduction to MCMC for Machine Learning*. Machine Learning 50:5-43, 2003. - [optional] Paper: Gareth O. Roberts and Jeffrey S. Rosenthal.
*Markov chain Monte Carlo*. Encyclopedia of the Actuarial Sciences, 2004. - [optional] Video: Iain Murray -- Markov Chain Monte Carlo
- [optional] Video: Nando de Freitas -- Monte Carlo Simulation for Statistical Inference
- [optional] Video: Christian Robert -- Markov Chain Monte Carlo Methods
- [optional] Slides: Jeffrey Rosenthal -- Understanding MCMC
- [optional] Metacademy: Markov Chain Monte Carlo
- [optional] Metacademy: Metropolis-Hastings
- [optional] Metacademy: Gibbs Sampling

Wed 30 Oct 2013

Lecture 15: Advanced Markov Chain Monte Carlo [ quiz ]

- [required] Book: Murphy -- Chapter 24, Sections 24.5-24.7 --
*Markov Chain Monte Carlo (MCMC) Inference* - [required] Paper: Radford M. Neal.
*Slice sampling (with discussion)*. Annals of Statistics 31:705-767, 2003. - [required] Paper: Radford M. Neal.
*MCMC using Hamiltonian dynamics*. Handbook of Markov Chain Monte Carlo, 2010. - [optional] Book: Bishop -- Chapter 11, Section 11.4-11.6 --
*Sampling Methods* - [optional] Book: MacKay -- Chapter 32 --
*Exact Monte Carlo Sampling* - [optional] Slides: Peter Green -- Trans-dimensional Markov chain Monte Carlo
- [optional] Paper: David I. Hastie and Peter J. Green -- Model choice using reversible jump Markov chain Monte Carlo
- [optional] Metacademy: Slice Sampling
- [optional] Metacademy: Hamiltonian Monte Carlo

Fri 1 Nov 2013

Section 8: MCMC Practicalities

- Final Project Proposal Feedback Available
- Midterms Back

Lecture 16: Latent Dirichlet Allocation [ quiz ]

- [required] Book: Murphy -- Chapter 27, Sections 27.1-27.4 --
*Latent Variable Models for Discrete Data* - [required] Paper: David M. Blei, Andrew Ng and Michael I. Jordan.
*Latent Dirichlet allocation*. Journal of Machine Learning Research 3:993-1022, 2003. - [required] Paper: David M. Blei and John D. Lafferty.
*Topic Models*. Text Mining: Classification, Clustering and Applications, 2009. - [required] Paper: Thomas L. Griffiths and Mark Steyvers.
*Finding Scientific Topics*. Proceedings of the National Academy of Sciences 101:5228-5235, 2004. - [optional] Paper: Thomas Hofmann.
*Probabilistic Latent Semantic Analysis*. UAI 1999. - [optional] Paper: Asuncion, Smyth and Welling -- Asynchronous Distributed Learning of Topic Models
- [optional] Video: David Blei -- Topic Models
- [optional] Slides: David Blei -- Probabilistic Topic Models
- [optional] Metacademy: Latent Dirichlet Allocation

Wed 6 Nov 2013

Lecture 17: State Space Models

- Assignment 4 Help Session 5-7pm, Maxwell-Dworkin Second Floor Lounge
- [required] Book: Murphy -- Chapter 18, Sections 18.1-18.4 --
*State Space Models* - [optional] Book: Murphy -- Chapter 18, Sections 18.5-18.6 --
*State Space Models* - [optional] Book: Bishop -- Chapter 13, Sections 13.3 --
*Sequential Data* - [optional] Paper: Eric A. Wan and Rudolph van der Merwe.
*The Unscented Kalman Filter for Nonlinear Estimation*. - [optional] Metacademy: Linear Dynamical Systems

Fri 8 Nov 2013

Section 9: Graph Models [ notes ]

- Assignment 4 Due
- Assignment 5 Out
- Last Day for Assignment 3 Regrades
- [required] Book: Murphy -- Chapter 27, Sections 27.5-27.6 --
*Latent Variable Models for Discrete Data*

Lecture 18: Kernels [ quiz ]

- [required] Book: Murphy -- Chapter 14 --
*Kernels* - [optional] Book: Bishop -- Chapter 6, Sections 6.1-6.2 --
*Kernel Methods* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 5 --
*Basis Expansions and Regularization* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 12 --
*Support Vector Machines and Flexibile Discriminants* - [optional] Slides: Andrew Moore -- Support Vector Machines
- [optional] Video: Bernhard Scholkopf -- Kernel Methods
- [optional] Video: Liva Ralaivola -- Introduction to Kernel Methods
- [optional] Video: Colin Campbell -- Introduction to Support Vector Machines
- [optional] Video: Alex Smola -- Kernel Methods and Support Vector Machines
- [optional] Video: Partha Niyogi -- Introduction to Kernel Methods
- [optional] Many more videos on kernel-related topics here
- [optional] Metacademy: The Kernel Trick
- [optional] Metacademy: Constructing Kernels
- [optional] Metacademy: Support Vector Machines

Wed 13 Nov 2013

Lecture 19: Gaussian Processes [ quiz ]

- [required] Book: Murphy -- Chapter 15 --
*Gaussian Processes* - [optional] Book: Rasmussen and Williams -- Chapter 2 --
*Regression* - [optional] Book: Rasmussen and Williams -- Chapter 3 --
*Classification* - [optional] Book: Bishop -- Chapter 6, Sections 6.4 --
*Kernel Methods* - [optional] Book: MacKay -- Chapter 45 --
*Gaussian Processes* - [required] Video: David MacKay -- Gaussian Process Basics
- [optional] Video: Carl Rasmussen -- Learning with Gaussian Processes
- [optional] Metacademy: Gaussian Process

Fri 15 Nov 2013

Section 10: Practical Gaussian Processes

- Assignment 4 Back
- Last Day for Midterm Regrades
- [required] Paper: Iain Murray, Ryan Prescott Adams and David J.C. MacKay.
*Elliptical Slice Sampling*. JMLR Workshop and Conference Proceedings 9:541-548, 2010. - [optional] Paper: Iain Murray and Ryan Prescott Adams.
*Slice Sampling Covariance Hyperparameters of Latent Gaussian Models*. Advances in Neural Information Processing Systems, 2011. - [optional] Book: Rasmussen and Williams -- Chapter 4 --
*Covariance Functions* - [optional] Book: Rasmussen and Williams -- Chapter 5 --
*Model Selection and Adaptation of Hyperparameters* - [optional] Video: Yee Whye Teh -- Nonparametric Bayesian Methods
- [optional] Video: Peter Orbanz -- Foundations of Nonparametric Bayesian Methods
- [optional] Video: Michael Jordan -- Dirichlet Processes, Chinese Restaurant Processes and All That
- [optional] Video: Yee Whye Teh -- Dirichlet Processes: Tutorial and Practical Course
- [optional] Video: Volker Tresp -- Dirichlet Processes and Nonparametric Bayesian Modelling
- [optional] Video: Zoubin Ghahramani -- Should All Machine Learning Be Bayesian? Should All Bayesian Models Be Nonparametric?

Mon 18 Nov 2013

Lecture 20: Dirichlet Processes I [ quiz ]

- [required] Book: Murphy -- Chapter 25, Sections 25.1-25.2 --
*Clustering* - [required] Paper: Peter Orbanz and Yee Whye Teh.
*Bayesian Nonparametric Models*. Encyclopedia of Machine Learning, 2010. - [required] Paper: Carl Edward Rasmussen.
*The Infinite Gaussian Mixture Model*. NIPS, 1999. - [optional] Video: Tom Griffiths -- Inferring Structure from Data
- [optional] Metacademy: Dirichlet Process
- [optional] Metacademy: Chinese Restaurant Process

Wed 20 Nov 2013

Lecture 21: Dirichlet Processes II [ quiz ]

- Assignment 5 Help Session 5-7pm, Maxwell-Dworkin Second Floor Lounge
- [optional] Paper: Jayaram Sethuraman.
*A Constructive Definition of Dirichlet Priors*. Statistica Sinica 4, 1994. - [optional] Paper: Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David M. Blei -- Hierarchical Dirichlet Processes, Journal of the American Statistical Association, 2006.
- [optional] Paper: Radford M. Neal -- Defining Priors for Distributions Using Dirichlet Diffusion Trees, Bayesian Statistics 7, 619-629, 2003.
- [optional] Paper: Zoubin Ghahramani, Thomas L. Griffiths and Peter Sollich -- Bayesian nonparametric latent feature models
- [optional] Metacademy: Hierarchical Dirichlet Process

Fri 22 Nov 2013

Section 11: Practical Dirichlet Processes

- Final Project Abstract and Status Report Due
- Assignment 5 Due
- [required] Paper: Radford M. Neal -- Markov Chain Sampling Methods for Dirichlet Process Mixture Models

Lecture 22: Boltzmann Machines

- [required] Book: Murphy -- Chapter 27, Section 27.7 --
*Latent Variable Models for Discrete Data* - [required] Book: Murphy -- Chapter 28, Section 28.1 --
*Deep Learning* - [optional] Book: MacKay -- Chapter 31 --
*Ising Models* - [optional] Book: MacKay -- Chapter 42 --
*Hopfield Networks* - [optional] Book: MacKay -- Chapter 43 --
*Boltzmann Machines* - [optional] Video: Geoffrey Hinton -- Deep Belief Networks
- [optional] Video: Marcus Frean -- Restricted Boltzmann Machines and Deep Belief Networks
- [optional] Video: Geoffrey Hinton -- A Tutorial on Deep Learning
- [optional] Video: Yoshua Bengio and Yann LeCun -- Tutorial on Deep Learning Architectures
- [optional] Video: Yann LeCun -- Visual Perception with Deep Learning

Mon 2 Dec 2013

Lecture 23: Neural Networks [ quiz ]

- Last Day for Assignment 4 Regrades
- Assignment 5 Back
- [required] Book: Murphy -- Chapter 16, Section 16.5 --
*Adaptive Basis Function Models* - [optional] Book: Bishop -- Chapter 5 --
*Neural Networks* - [optional] Book: MacKay -- Chapter 39 --
*The Single Neuron as a Classifier* - [optional] Book: MacKay -- Chapter 40 --
*Capacity of a Single Neuron* - [optional] Book: MacKay -- Chapter 41 --
*Learning as Inference* - [optional] Book: MacKay -- Chapter 44 --
*Supervised Learning in Multilayer Networks* - [optional] Book: Hastie, Tibshirani, and Friedman -- Chapter 11 --
*Neural Networks* - [optional] Metacademy: Feedforward Neural Networks

Wed 4 Dec 2013

Lecture 24: Advanced Neural Networks [ quiz ]

- [required] Book: Murphy -- Chapter 28, Sections 28.3-28.5 --
*Deep Learning* - [optional] Paper: Geoffrey E. Hinton, Simon Osindero and Yee Whye Teh.
*A Fast Learning Algorithm for Deep Belief Nets*. Neural Computation 18:1527-1554, 2006. - [optional] Metacademy: Convolutional Neural Networks

Thu 5 Dec 2013

Wed 11 Dec 2013

- Final Project Poster Session

Wed 11 Dec 2013

- Final Project Reports Due
- Last Day for Assignment 5 Regrades

- Assignment 1: Out 6 September 2013; Due 20 September 2013 [ assignment-1.pdf | assignment-1.tex ]

**Jester Data:**These data are approximately 1.7 million ratings in the range [-10,10] of 150 jokes from 63,974 users. These data are from the Eigentaste Project at Berkeley. I have munged the data somewhat, so use the local copies here:- jester_ratings.dat.gz: Each row is formatted as [User ID] [Joke ID] [Rating]
- jester_items.dat.gz: Maps the joke IDs to the joke text.
- jester_items.clean.dat.gz: Maps the joke IDs to the joke text. (No punctuation, lowercased.)
- jester_ratings_users.mat: Ratings chopped up into a cell array with one entry per user.
- jester_ratings_jokes.mat: Ratings chopped up into a cell array with one entry per joke.

*Eigentaste: A Constant Time Collaborative Filtering Algorithm*. Information Retrieval 4(2):133-151, 2001. [pdf] - Assignment 2: Out 21 September 2013; Due 4 October 2013 [ assignment-2.pdf | assignment-2.tex ]

**Spam Email Data:**These data are 3000 training and 1601 test emails. Each has 57 features and a binary label in the last column. You can read more about the data here. - Assignment 3: Out 4 October 2013; Due
~~18 October 2013~~20 October 2013 [ assignment-3.pdf | assignment-3.tex ]

Uses Jester data above.

- Assignment 4: Out 25 October 2013; Due 8 November 2013 [ assignment-4.pdf | assignment-4.tex ]
- Assignment 5: Out 8 November 2013; Due 22 November 2013 [assignment-5.pdf | assignment-5.tex ]

- Proposal: Due 25 October 2013
- Abstract and Status Report: Due 22 November 2013
- Poster Session: 5 December 2013 (2-4pm in Maxwell-Dworking)
- Final Report: Due 11 December 2013

- Assignments (lowest dropped): 40%
- Pre-lecture Quizzes: 5%
- Midterm: 15%
- Project Proposal: 10%
- Project Status Report and Abstract: 5%
- Project Poster: 5%
- Project Report: 20%

**Required:**Kevin Murphy, Machine Learning: A Probabilistic Perspective, MIT Press.- Recommended: Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer.
- Optional: David J.C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press.
**Freely available online.** - Optional: Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, Springer.
**Freely available online.** - Optional: David Barber, Bayesian Reasoning and Machine Learning, Cambridge University Press.
**Freely available online.**

- Carl Edward Rasmussen and Christopher K.I. Williams,
*Gaussian Processes for Machine Learning*, MIT Press.**Freely available online.** - Luc Devroye,
*Non-Uniform Random Variate Generation*, Springer-Verlag.**Freely available online.** - Daphne Koller and Nir Friedman,
*Probabilistic Graphical Models: Principles and Techniques*, MIT Press. - Jorge Nocedal and Stephen J. Wright,
*Numerical Optimization*, Springer. - Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin.
*Bayesian Data Analysis*, CRC. - Thomas M. Cover and Joy A. Thomas,
*Elements of Information Theory*, Wiley. - Christian P. Robert and George Casella,
*Monte Carlo Statistical Methods*, Springer.

**What is****Metacademy?**Metacademy is an exciting new online tool developed by Roger Grosse and Colorado Reed for helping you to develop personalized instruction. It's meant to help you manage what you know about different topics and develop an individualized curriculum to learn a new subject.**I have an interview/sporting event/illness/computer crash. Can I have an extension?**No. You can turn your assignment in up to a week late for half credit, and your lowest assignment grade will be dropped.**Do I need to turn in code?**No. You will write up your results with graphs, tables, and descriptions.**How should I format my assignment?**Use the provided .tex and .cls files (see the examples) to produce a LaTeX document. Compile it to PDF and upload the result to the iSites dropbox.**What if I don't know how to use LaTeX?**Everyone doing quantitative work should know how to use LaTeX, so consider this class an opportunity to learn it.**Is attendance required at lecture/section?**No, attendance is not required. However, you will be assessed on material that is presented in both lecture and section and may or may not be available from the readings alone.**Can I do the final project without a partner?**Yes.**Can I have a group of three for the final project?**Yes, but you'll be expected to do an amount of work appropriate for three people. I suggest discussing this with the instructors.**I think I have found an error in the Murphy book!**This is entirely possible. Let's share a comprehensive list of these in Piazza.**What is your policy on simultaneous enrollment?**You may not be simultaneously enrolled in this class and another.