Deep Learning Theory (CS 598 DLT).

Essentials.

Schedule.

Schedule tentative! Check back often!

Date Topic Assignments
8/27 No lecture! hw0 handout, hw0 template
8/29 No lecture!
9/3 Introduction; initial constructive approximations
9/5 Constructive 3- and non-constructive 2-layer apx hw0 due
9/10 Infinite width 1: univariate, Maurey
9/12 Infinite width 2: Barron
9/17
9/19
9/24
Infinite width 3: Neural Tangent Kernel (NTK)
9/26
10/1
Benefits of depth
basic separations; sobolev spaces
10/3
10/8
10/10
10/15
Optimization basics
Smooth optimization, nondifferentiabilities and differential inclusion, stochastic optimization
hw1 handout, hw1 template
10/17
10/22
10/24
Shallow and deep network optimization
(Various vignettes)


project proposal out
10/29
10/31
11/5

11/7
Rademacher bounds for deep networks
For basics of concentration, see also my ml theory course and Martin Wainwright’s concentration notes


hw1 due
hw2 handout, hw2 template
project proposal due
11/12
11/14
Covering number bounds for deep networks
Omitted lectures: VC bounds for deep networks
11/19 [14] Philip Amortila
[28] John Anthony Pavlik
[8] Gao Tang
[1] Zihao Yang
[26] Linyi Li
[2] Erchi Wang
[6] Vasileios Livanos
11/21 [3] Ziwei Ji
[17] Lihui Liu
[23] Yibo Zhang
[18] Amnon Attali
[9] Tengyang Xie
[19] Junyeob Lim
[16] Eddie Huang
12/3 No lecture!
12/5 [24] Qinghai Zhou
[31] Efthymios Tzinis
[12] Si Zhang
[13] Gang Qiao
[10] Priyank Agrawal
[27] Shiliang Zuo
[20] Sahand Mozaffari
12/10 [15] Wentao Yang
[7] Mohit Vyas
[22] Dimitrios Gotsis
[11] Jian Kang
[29] Raphael Long
[21] Christian Howard
12/17 (No lecture.) hw2 due

Project.

The project is a paper presentation.