Machine learning in signal processing WS 20/21

This course is an introduction into statistical machine learning and artificial intelligence. The special emphasis is on applications to modern signal processing problems. The course is focused on design principles of machine learning algorithms.

First we will study basic methods for regression and classification: linear regression, logistic regression, the nearest neighbors algorithm. Based on these examples, we will discuss the fundamental trade-off between the flexibility of the model and the ability to fit the model based on the moderate amount of training data. We will contrast learning in high-dimensional spaces vs. learning in low dimensional spaces.

Next, we will study methods that help make linear models flexible: polynomial features and splines. When these tools are used, regularization is crucial. We will discuss structured signal representations: short-time Fourier transform and wavelets. We will focus on the importance of sparsity in signal representations.

This will lead us to compressed sensing and to other modern convex-optimization-based methods for signal denoising, reconstruction, and compression. We will review key concepts in convex optimization, study the LASSO, support vector machines, the idea of kernels.

The last part of the course will focus on the breakthrough new technology for computer vision: the deep learning.

The course contains exercises: 40 percent mathematical and 60 percent implementing basic algorithms in Python. You will be asked to implement basic machine learning and signal processing algorithms yourself. For more advanced algorithms, you will practice using powerful numerical and optimization libraries (numpy, cvxpy, scikit-learn, pywavelets, pytorch).

Database info is here.

Learning goals are here.

Literature

We follow:

  • T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition. (link)  —  Chapters 1–7.

  • A. Ng: Lecture notes and materials for Stanford CS229 class. (link)  —  Lecture Notes and Exercises.

  • M. Kon: Lecture notes on basics of wavelets. (link)

  • M. Nielsen: Neural networks and deep learning. (link)

Basic online course on probability and statistics:

  • J. Orloff and J. Bloom. 18.05 Introduction to probability and statistics. Spring 2014. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA. (link)

Basic online course and books on linear algebra:

  • G. Strang. 18.06 Linear Algebra. Spring 2010. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.

  • G. Strang: Introduction to linear algebra.

  • G. Strang: Linear algebra and learning from data.

Suggested next steps:

  • C. M. Bishop: Pattern recognition and machine learning.

  • S. Mallat: A wavelet tour of signal processing, Third edition: The sparse way.

  • C. M. Bishop: Neural networks for pattern recognition.

  • I. Goodfellow, Y. Bengio, A. Courville: Deep learning. (link)

  • E. Candes: Modern statistical estimation via oracle inequalities. (link)

  • D. Donoho, H. Monajemi, V. Papyan: Theories of deep learning class Stanford STATS385 class. (link)

  • C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals: Understanding deep learning requires rethinking generalization. (link)

  • S. Shalev-Shwartz and S. Ben-David: Understanding machine learning: from theory to algorithms.

Time

Lectures:

  • Pre-recorded video lectures will be published online on this website. You will need to be logged into StudOn to access the lectures. There will be NO live video lectures. The video lectures will appear no later than when a normal lecture would have happened. You might need to use the refresh button in your browser to see the latest updates to this page.

Review sessions:

  • First two review sessions will be pre-recorded and published online on this website. Announcement on how the remaining 4 review sessions will be conducted, will be made later.

Office hours:

  • There will be office hours conducted via Zoom starting in the second week of the semester. You will get information via StudOn with time and access details.

Communication

To be kept up to date, please register for the course on StudOn.

Lecture handouts and videos

Introduction

Feature modeling

Convex optimization-based methods

Neural networks

Discussion handouts

Problem sets

Problem set Solution
Problem set 1 Solution to PS1
Problem set 2 Solution to PS2
Problem set 3 Solution to PS3
Problem set 4 Solution to PS4
Problem set 5 Solution to PS5