Machine Learning Theory 2018 [ CS 598 Tel ]

Essentials

Schedule

Date. Topics. Notes. Coursework.
8/27 Overview. pdf. hw0 out: tex, pdf.
Representation/approximation.
8/29 Overview; failure of linear. pdf.
9/3 No class: labor day! hw0 due!
9/5 Box apx (linear over boxes; decision trees). pdf.
9/10 Box apx (3 layer networks). pdf.
9/12 Poly apx (RBF SVM, 2-layer networks). pdf.
9/17 Succinct 1 (start of depth separation). pdf.
9/19 Succinct 2 (end of depth separation). pdf. hw1 out: tex, pdf. hw1v2: tex, pdf, diff. hw1v3: tex, pdf, diff.
9/24 Succinct 3 (networks for squaring and distributions). pdf.
Optimization & online learning.
9/26 Online learning 1: Perceptron. pdf.
10/1 Online learning 2: concept learning. pdf.
10/3 Batch optimization 1: smooth. pdf.
10/8 Batch optimization 2: strongly convex; lipschitz. pdf. hw1 due.
10/10 Maurey sparsification and Frank-Wolfe. pdf.
10/15 Convex risk minimization and classification. pdf.
10/17 No class. “Fun reading”: my old convexity notes: pdf, pdf.
10/22 Approximate gradients. pdf.
10/24 Optimization summary and open problems. pdf.
Generalization.
10/29 Concentration of measure. pdf. project proposal: tex, bib, pdf.
10/31 Finite classes and primitive covers. pdf.
11/5 Symmetrization and Rademacher complexity. pdf. hw2 out: tex, pdf. hw2v2: tex, pdf, diff.
11/7 No class!
11/12 Properties of Rademacher complexity. pdf.
11/14 Classification bounds. pdf. proposal due; proposal meetings.
11/26 VC dimension or linear functions and linear threshold networks. pdf.
11/28 VC dimension of ReLU networks. pdf. hw2 due.
12/3 Covering numbers and Rademacher complexity. pdf. hw3 out: tex, pdf.
12/5 Covering and Rademacher bounds for neural networks. pdf.
Miscellaneous
12/10 Nearest neighbor. pdf.
12/12 Fast rates. pdf.
Final presentations and homework.
12/13 Reading day: project presentations 1-4pm!

Homework policies

Project policies

Resources

Other learning theory-ish classes. All of these courses are different, and all have good material, and there are many I neglected to include!

Textbooks and surveys. Again, there are many others, but here are a key few.