Transfer learning

Convolutional neural network (CNN)

Computer Vision

Graph neural network (GNN)

Recurrent neural network (RNN)

Long Short-Term Memory (LSTM)

Sequence-to-sequence (Seq2Seq)

Attention (machine learning)

Self-supervision

Autoencoders

Transformer (machine learning model)

Meta-learning

Fine-tuning

Generative Models

CS 182/282A: Deep Neural Networks UC Berkeley Fall 2022 An advanced course dealing with deep networks in the fields of computer vision, language technology, robotics, and control. It delves into the themes of deep learning, model families, and real-world applications. A strong mathematical background in calculus, linear algebra, probability, optimization, and statistical learning is necessary. Deep Networks have revolutionized computer vision, language technology, robotics and control. They have a growing impact in many other areas of science and engineering, and increasingly, on commerce and society. They do not however, follow any currently known compact set of theoretical principles. In Yann Lecun's words they require "an interplay between intuitive insights, theoretical modeling, practical implementations, empirical studies, and scientific analyses." This is a fancy way of saying “we don’t understand this stuff nearly well enough, but we have no choice but to muddle through anyway.” This course attempts to cover that ground and show you how to muddle through even as we aspire to do more. The goal is to teach a principled course in Deep Learning that serves the diverse needs of our students while also codifying the present understanding of the field. Topics covered may include, but are not limited to:
- Underlying themes of deep learning, including building beyond underlying machine learning concepts like supervised vs unsupervised learning, regression and classification, training/validation/testing, distribution shifts, regularization, the fundamental underlying tradeoffs;
- Defining and training neural networks: features, computation graphs, backpropagation, iterative optimization (SGD, Newton’s Method, Momentum, RMSProp, AdaGrad, Adam), strategies for training (explicit and implicit regularization, batch and layer normalization, weight initialization, gradient clipping, ensembles, dropout), hyperparameter tuning
- Families of contemporary models: fully connected networks, convolutional nets, graph neural nets, recurrent neural nets, transformers
- Problems that utilize neural networks: computer vision, natural language processing, generative models, and others.
- Conducting experiments in a systematic, repeatable way, leveraging and presenting data from experiments to reason about network behavior. This is a graduate-level/advanced undergraduate course about a particular approach to information processing using (simulated) analog circuits where the desired circuit behavior is tuned via optimization involving data since we have no idea how to do hand-tuning at scale. Probabilistic frames are useful to understand what is going on, as well as how we navigate certain design choices. Overall, we expect students to have a strong mathematical background in calculus, linear algebra, probability, optimization, and statistical learning. Berkeley undergraduate courses that can help build maturity include:
- **Calculus**: Math 53 (note: Math 1B or AP Math is not enough)
- **Linear Algebra and Optimization**: EECS 16B and EECS 127/227A is ideal, but EECS 16B alone might be enough if students have complete mastery of that material. Math 110 is also helpful. (note: Math 54 or EECS 16A is required as a minimum, but are not nearly enough.)
- **Probability**: EECS 126, Stat 134, or Stat 140 (note: CS 70 is required at a minimum, but might not be enough for everyone)
- **Statistical Learning**: CS 189/289A or Stat 154 (note: Data 102 is insufficient, even when combined with Data 100.)

*Math 53 and EECS 126 and EECS 127 and CS 189 is the recommended background.*

Prerequisites are not enforced for enrollment, but we encourage you to consider taking some of the classes listed above and save this course for a future semester if you feel shaky on the fundamentals.

The course assumes familiarity with programming in a high-level language with data structures. Homeworks and projects will typically use Python. We encourage you to check out this [tutorial](https://docs.python.org/3/tutorial/) if you haven’t used it before. Students who have taken Berkeley courses like CS 61A and CS 61B are well-prepared for the programming components of the class.

We do not have the staff bandwidth to help students with material that they should have understood before taking this course. If you choose to proceed with this course, you are accepting full responsibility to teach yourself anything in your background that you are missing. We will not be slowing down to accommodate you, and questions pertaining to background material will always have the lowest priority in all course forums. 

CS 182/282A: Deep Neural Networks

Deep Learning

Neural network

Python

NumPy

Logistic Regression

Batch Normalization

Generative adversarial network (GAN)

Shallow Neural Network

Deep Neural Network

Optimization Algorithms

Initialization

Regularization (mathematics)

Gradient Checking

Optimization

Hyperparameter Tuning

TensorFlow

Deep Convolutional Models

Residual Networks

Detection Algorithms

Face Recognition

Neural Style Transfer

You Only Look Once (YOLO)

ConvNets

Image Segmentation

U-Net

Natural Language Processing (NLP)

Word Embeddings

Deep Reinforcement Learning

Neural Machine Translation

Trigger Word Detection

CS 230 Deep Learning Stanford University Fall 2022 An in-depth course focused on building neural networks and leading successful machine learning projects. It covers Convolutional Networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more. Students are expected to have basic computer science skills, probability theory knowledge, and linear algebra familiarity.  Deep Learning is one of the most highly sought after skills in AI. In this course, you will learn the foundations of Deep Learning, understand how to build neural networks, and learn how to lead successful machine learning projects. You will learn about Convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more.  Students are expected to have the following background, and are invited to take the [Workera](http://www.workera.ai/) technical assessments prior to the class to self-assess themselves prior to taking the class:

- Knowledge of basic computer science principles and skills, at a level sufficient to write a reasonably non-trivial computer program. This corresponds to a Developing level (or badge) in the “Algorithmic Coding” section on [Workera](http://www.workera.ai/).
- Familiarity with the probability theory (CS 109 or STATS 116), which students can assess by taking the “Data Science” section on [Workera](http://www.workera.ai/).
- Familiarity with linear algebra (MATH 51), which students can assess by taking the “Mathematics” section on [Workera](http://www.workera.ai/). 

CS 230 Deep Learning

Multi-task Learning

PyTorch

Black-box Meta-learning

In-context Learning

Optimization-based Meta-learning

Non-Parametric Few-Shot Learning

Unsupervised pre-training

Task Construction

Large-scale Meta-optimization

Variational Inference

Bayesian Meta-learning

Domain Adaptation

Domain Generalization

Lifelong Learning

CS 330 Deep Multi-Task and Meta Learning Stanford University Fall 2022 This course emphasizes leveraging shared structures in multiple tasks to enhance learning efficiency in deep learning. It provides a thorough understanding of multi-task and meta-learning algorithms with a focus on topics like self-supervised pre-training, few-shot learning, and lifelong learning. Prerequisites include an introductory machine learning course. The course is designed for graduate-level students. While deep learning has achieved remarkable success in many problems such as image classification, natural language processing, and speech recognition, these models are, to a large degree, specialized for the single task they are trained for. This course will cover the setting where there are multiple tasks to be solved, and study how the structure arising from multiple tasks can be leveraged to learn more efficiently or effectively. This includes:

- self-supervised pre-training for downstream few-shot learning and transfer learning
- meta-learning methods that aim to learn efficient learning algorithms that can learn new tasks quickly
- curriculum and lifelong learning, where the problem requires learning a sequence of tasks, leveraging their shared structure to enable knowledge transfer

This is a graduate-level course. By the end of the course, students will be able to understand and implement the state-of-the-art multi-task learning and meta-learning algorithms and be ready to conduct research on these topics.  CS 229 or an equivalent introductory machine learning course is required. 

CS 330 Deep Multi-Task and Meta Learning

K-Nearest Neighbors

Linear Classifiers

Stochastic gradient descent (SGD)

Momentum

AdaGrad

Adam

Learning Rate Schedules

Support Vector Machine (SVM)

Softmax Loss

Multi-layer Perceptron

Backpropagation

Higher-level representations

Image Features

Pooling

AlexNet, VGG, GoogLeNet, ResNet

Activation Functions

Data Processing

Weight initialization

Data Augmentation

Feature visualization and inversion

Adversarial Examples

DeepDream and Style Transfer

Single-stage detectors

Two-stage detectors

Semantic/Instance/Panoptic segmentation

Gated recurrent unit (GRU)

Language Modeling

Image Captioning

Object Detection

Self-Attention

Video classification

3D CNNs

Two-stream networks

Multimodal Video Understanding

Supervised learning

Unsupervised learning

Pixel RNN, Pixel CNN

Variational autoencoder (VAE)

Pretext Tasks

Contrastive Learning

Multisensory Supervision

Optical Flow

Stereo Vision

3D Shape Representations

Shape Reconstruction

Neural Implicit Representations

Convolution

CS231n: Deep Learning for Computer Vision Stanford University Spring 2022 This is a deep-dive into the details of deep learning architectures for visual recognition tasks. The course provides students with the ability to implement, train their own neural networks and understand state-of-the-art computer vision research. It requires Python proficiency and familiarity with calculus, linear algebra, probability, and statistics. Computer Vision has become ubiquitous in our society, with applications in search, image understanding, apps, mapping, medicine, drones, and self-driving cars. Core to many of these applications are visual recognition tasks such as image classification, localization and detection. Recent developments in neural network (aka “deep learning”) approaches have greatly advanced the performance of these state-of-the-art visual recognition systems. This course is a deep dive into the details of deep learning architectures with a focus on learning end-to-end models for these tasks, particularly image classification. During the 10-week course, students will learn to implement and train their own neural networks and gain a detailed understanding of cutting-edge research in computer vision. Additionally, the final assignment will give them the opportunity to train and apply multi-million parameter networks on real-world vision problems of their choice. Through multiple hands-on assignments and the final course project, students will acquire the toolset for setting up deep learning tasks and practical engineering tricks for training and fine-tuning deep neural networks.  - Proficiency in Python  
    All class assignments will be in Python (and use numpy) (we provide a tutorial [here](http://cs231n.github.io/python-numpy-tutorial/) for those who aren't as familiar with Python). If you have a lot of programming experience but in a different language (e.g. C/C++/Matlab/Javascript) you will probably be fine.
- College Calculus, Linear Algebra (e.g. MATH 19 or 41, MATH 51)  
     You should be comfortable taking derivatives and understanding matrix vector operations and notation.
- Basic Probability and Statistics (e.g. CS 109 or other stats course)  
    You should know basics of probabilities, gaussian distributions, mean, standard deviation, etc. 

CS231n: Deep Learning for Computer Vision

Transfer learning

4 courses cover this concept

CS 182/282A: Deep Neural Networks

CS 230 Deep Learning

CS 330 Deep Multi-Task and Meta Learning

CS231n: Deep Learning for Computer Vision