DLT project.

Grading summary.

You may work in groups of at most two, but you may discuss with anyone in the class as much as you like (see below).

To receive more than zero points:

• Project phase 1 spreadsheet post before Friday. November 12, 11:59pm.

• Project phase 1 spreadsheet haggling finished before Friday, November 19, 11:59pm.

• Project phase 2 gradescope submission before Wednesday, December 1, 11:59pm.

Phase 1.

This phase is to select a paper. Here’s the process:

1. I will post a class-wide spreadsheet before the end of Friday, November 5.

   • Here is a link to the spreadsheet, which is accessible with @illinois email accounts.

   • To access the spreadsheet, you must have google authentication set to use your illinois account. I will not grant any individual access requests (e.g., to gmail accounts).

   • To enable google authentication on your illinois account, see the instructions at this link, alternatively it probably suffices to enable “google apps @ illinois” at this link.

2. Each group grabs one row of the spreadsheet, first come first serve.

3. You will then propose one paper at a time, waiting for my comments before posting another.

   • Your post must include paper title, paper authors, and a URL to a freely-available version of the paper (no paywalls).

   • I will check the spreadsheet every few fays between November 5 and 19, either rejecting or accepting papers I see.

   • When rejecting, I will mark the paper red, and leave a comment. Reasons I may reject: paper lacks sufficient theory, paper lacks sufficient relevance to the class, conversely the paper is too close to a lecture topic (e.g., we covered it explicitly), paper is older than July 1, 2018, paper was already chosen by another group… However, the paper does not need to be accepted/published. After I reject, you should start a new spreadsheet cell to the right and we try again.

   • When accepting, I’ll color the paper green.

Here are some example acceptable papers, and how I’d like them formatted for the spreadsheet. You do not need to restrict your choices to this list, these are just examples.

• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss.
  Lenaic Chizat, Francis Bach.
  https://arxiv.org/abs/2002.04486

• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data.
  Colin Wei, Kendrick Shen, Yining Chen, Tengyu Ma.
  https://arxiv.org/abs/2010.03622

• Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers.
  Colin Wei, Yining Chen, Tengyu Ma.
  https://arxiv.org/abs/2107.13163

• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel.
  Colin Wei, Jason D. Lee, Qiang Liu, Tengyu Ma.
  https://arxiv.org/abs/1810.05369

• Hardness of Learning Neural Networks with Natural Weights.
  Amit Daniely, Gal Vardi.
  https://arxiv.org/abs/2006.03177v2

• Neural Networks with Small Weights and Depth-Separation Barriers.
  Gal Vardi, Ohad Shamir.
  https://arxiv.org/abs/2006.00625v3

• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network.
  Taiji Suzuki, Hiroshi Abe, Tomoaki Nishimura.
  https://arxiv.org/abs/1909.11274v3

• Deep Equals Shallow for ReLU Networks in Kernel Regimes.
  Alberto Bietti, Francis Bach.
  https://arxiv.org/abs/2009.14397

• Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias.
  Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, Sanjeev Arora.
  https://arxiv.org/abs/2110.13905

Phase 2.

• You will submit a typed (\LaTeX) 2-page paper summary on gradescope, with additional pages for references. I will not read past those 2 body pages, so don’t put key stuff afterwards (if you choose to submit something longer).

• Here is a \LaTeX template, and here is the corresponding compiled PDF. Please use this template.

• I will give full credit to handins which use the paper I selected and follow the sections of the template and fill them in reasonably. When in doubt, simply try to make the project useful to you; i.e., a summary of the paper which you’d find useful to refer back to later.

Paper discussion and academic integrity.

• You may discuss with anyone in the class, not just your project partner. I have created a discord channel.

• You may not discuss in detail with people outside the class.

• We will have a purely optional in-class discussion of all papers as a final lecture. You do not need to show up, the project will have been graded by then.