Mathematical methods for machine learning and signal processing SS 19

This course focuses on modern machine learning and signal processing algorithms that have firm mathematical footing.

First, we will study the basics of frame theory – a mathematical framework for linear redundant signal expansions. We will discuss an application in signal sampling.

Next, we will study the theory of compressed sensing – a powerful way to recover sparse signals from an incomplete set of measurements. We will discuss applications in MRI and in super-resolution microscopy. Further, we will discuss matrix completion, an application of compressed sensing-based methodology in machine learning.

In the last part of the course we will focus on audio and image recognition problems. We will discuss two classical algorithms that achieved state-of-the-art performance before the deep learning revolution: Shazam for audio recognition and SIFT for image recognition. Finally, we will study the theory of scattering transform – a signal representation based on deep neural network that is invariant to signal translations and deformations. This topic is one of the few mathematical results related to theoretical understanding of deep learning.

Database info is here.

Learning goals

Students are able to:

• Theoretically analyze modern Machine Learning and Signal Processing algorithms.

• Develop new state-of-the-art algorithms.

• Do research in the field of modern Machine Learning and Signal Processing.

Literature

• V. I. Morgenshtern and H. Bölcskei: A short course on frame theory.

• S. Foucart and H. Rauhut: A mathematical introduction to compressive sensing.

• V. I. Morgenshtern and E. J. Candès: Super-resolution of positive sources: the discrete setup.

• D. Lowe: Distinctive image features from scale-invariant keypoints.

• A. Wang: An industrial-strength audio search algorithm.

• J. Bruna: Scattering representations for recognition.

• J. Bruna and S. Mallat: Invariant scattering convolution networks.

Other useful resources

• O. Christensen: An introduction to frames and Riesz bases.

• K. Gröchenig: Foundations of time-frequency analysis.

• S. Mallat: A wavelet tour of signal processing, third Edition: the sparse way.

• I. Daubechies: Ten lectures on wavelets.

• E. J. Candès, J. Romberg and T. Tao: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information.

• D. Donoho: Compressed sensing.

• E. Candes: Modern statistical estimation via oracle inequalities. (link)

• I. Goodfellow, Y. Bengio, A. Courville: Deep learning. (link)

• D. Donoho, H. Monajemi, V. Papyan: Theories of deep learning class Stanford STATS385 class. (link)

Time

Lectures:

• Tuesday, 8:15 - 9:45, Cauerstraße 7/9, 05.025

• Thursday, 12:15 - 13:45, Cauerstraße 7/9, 05.025

Review sessions:

• Thursday, 16:15 - 17:45, Cauerstraße 7/9, 05.025

Communication

To be kept up to date, please register for the course on StudOn. The password will be provided in the lecture.

Lecture handouts

Frame theory

• Lectures 1-7: A short course on frame theory.

Compressed sensing

• Lectures 8-10: Deterministic results.

• Lecture 11: Matching pursuit and orthogonal matching pursuit algorithms.

• Lectures 12-13: Super-resolution of positive sources.

• Lectures 14-17: Randomized sampling.

• Lecture 18: Principle component analysis.

• Lectures 19-20: Matrix completion.

Scattering transform

• Lecture 21: Scattering transform.

Discussion handouts

• Discussions 1 and 2: Review of Hilbert spaces and linear operators by E. Riegler.

• Discussion 3: Solution to problem set 1, problems 4, 5, 6, and 2.

• Discussion 4: Solution to problem set 2, problems 1.1, 2, 7, 6, and 8.

• Discussion 5: Solution to problem set 3, problems 8, 7, 6, 4, 5.

• Discussion 6: Scattering transform. Solution to problem set 4, problem 5.

• Discussion 7: Solution to problem set 5. Exam preparation.

Problem sets

Exam

Exam topics.