Convolutional neural network (CNN)

Computer Vision

Graph neural network (GNN)

Recurrent neural network (RNN)

Long Short-Term Memory (LSTM)

Sequence-to-sequence (Seq2Seq)

Attention (machine learning)

Self-supervision

Autoencoders

Transformer (machine learning model)

Meta-learning

Fine-tuning

Generative Models

Transfer learning

CS 182/282A: Deep Neural Networks UC Berkeley Fall 2022 An advanced course dealing with deep networks in the fields of computer vision, language technology, robotics, and control. It delves into the themes of deep learning, model families, and real-world applications. A strong mathematical background in calculus, linear algebra, probability, optimization, and statistical learning is necessary. Deep Networks have revolutionized computer vision, language technology, robotics and control. They have a growing impact in many other areas of science and engineering, and increasingly, on commerce and society. They do not however, follow any currently known compact set of theoretical principles. In Yann Lecun's words they require "an interplay between intuitive insights, theoretical modeling, practical implementations, empirical studies, and scientific analyses." This is a fancy way of saying “we don’t understand this stuff nearly well enough, but we have no choice but to muddle through anyway.” This course attempts to cover that ground and show you how to muddle through even as we aspire to do more. The goal is to teach a principled course in Deep Learning that serves the diverse needs of our students while also codifying the present understanding of the field. Topics covered may include, but are not limited to:
- Underlying themes of deep learning, including building beyond underlying machine learning concepts like supervised vs unsupervised learning, regression and classification, training/validation/testing, distribution shifts, regularization, the fundamental underlying tradeoffs;
- Defining and training neural networks: features, computation graphs, backpropagation, iterative optimization (SGD, Newton’s Method, Momentum, RMSProp, AdaGrad, Adam), strategies for training (explicit and implicit regularization, batch and layer normalization, weight initialization, gradient clipping, ensembles, dropout), hyperparameter tuning
- Families of contemporary models: fully connected networks, convolutional nets, graph neural nets, recurrent neural nets, transformers
- Problems that utilize neural networks: computer vision, natural language processing, generative models, and others.
- Conducting experiments in a systematic, repeatable way, leveraging and presenting data from experiments to reason about network behavior. This is a graduate-level/advanced undergraduate course about a particular approach to information processing using (simulated) analog circuits where the desired circuit behavior is tuned via optimization involving data since we have no idea how to do hand-tuning at scale. Probabilistic frames are useful to understand what is going on, as well as how we navigate certain design choices. Overall, we expect students to have a strong mathematical background in calculus, linear algebra, probability, optimization, and statistical learning. Berkeley undergraduate courses that can help build maturity include:
- **Calculus**: Math 53 (note: Math 1B or AP Math is not enough)
- **Linear Algebra and Optimization**: EECS 16B and EECS 127/227A is ideal, but EECS 16B alone might be enough if students have complete mastery of that material. Math 110 is also helpful. (note: Math 54 or EECS 16A is required as a minimum, but are not nearly enough.)
- **Probability**: EECS 126, Stat 134, or Stat 140 (note: CS 70 is required at a minimum, but might not be enough for everyone)
- **Statistical Learning**: CS 189/289A or Stat 154 (note: Data 102 is insufficient, even when combined with Data 100.)

*Math 53 and EECS 126 and EECS 127 and CS 189 is the recommended background.*

Prerequisites are not enforced for enrollment, but we encourage you to consider taking some of the classes listed above and save this course for a future semester if you feel shaky on the fundamentals.

The course assumes familiarity with programming in a high-level language with data structures. Homeworks and projects will typically use Python. We encourage you to check out this [tutorial](https://docs.python.org/3/tutorial/) if you haven’t used it before. Students who have taken Berkeley courses like CS 61A and CS 61B are well-prepared for the programming components of the class.

We do not have the staff bandwidth to help students with material that they should have understood before taking this course. If you choose to proceed with this course, you are accepting full responsibility to teach yourself anything in your background that you are missing. We will not be slowing down to accommodate you, and questions pertaining to background material will always have the lowest priority in all course forums. 

CS 182/282A: Deep Neural Networks

Deep Learning

Word Vectors

Word Window Classification

Language Models

Backpropagation

Neural network

Dependency Parsing

Subword Models

Self-Attention

Natural Language Generation

Pretraining

Reinforcement Learning from Human Feedback (RLHF)

Question Answering (QA)

Tree Recursive Neural Networks

Constituency Parsing

Code Generation

Multimodal Deep Learning

Coreference Resolution

CS 224N: Natural Language Processing with Deep Learning Stanford University Winter 2023 CS 224N provides an in-depth introduction to neural networks for NLP, focusing on end-to-end neural models. The course covers topics such as word vectors, recurrent neural networks, and transformer models, among others. Natural language processing (NLP) is a crucial part of artificial intelligence (AI), modeling how people share information. In recent years, deep learning approaches have obtained very high performance on many NLP tasks. In this course, students gain a thorough introduction to cutting-edge neural networks for NLP. ## What is this course about?

Natural language processing (NLP) or computational linguistics is one of the most important technologies of the information age. Applications of NLP are everywhere because people communicate almost everything in language: web search, advertising, emails, customer service, language translation, virtual agents, medical reports, politics, etc. In the last decade, deep learning (or neural network) approaches have obtained very high performance across many different NLP tasks, using single end-to-end neural models that do not require traditional, task-specific feature engineering. In this course, students will gain a thorough introduction to cutting-edge research in Deep Learning for NLP. Through lectures, assignments and a final project, students will learn the necessary skills to design, implement, and understand their own neural network models, using the Pytorch framework.

>  “Take it. CS221 taught me algorithms. CS229 taught me math. CS224N taught me how to write machine learning models.” – A CS224N student on Carta - **Proficiency in Python**
    All class assignments will be in Python (using NumPy and PyTorch). If you need to remind yourself of Python, or you're not very familiar with NumPy, you can come to the Python review session in week 1 (listed in the schedule). If you have a lot of programming experience but in a different language (e.g. C/C++/Matlab/Java/Javascript), you will probably be fine.
- **College Calculus, Linear Algebra** (e.g. MATH 51, CME 100)
    You should be comfortable taking (multivariable) derivatives and understanding matrix/vector notation and operations.

- **Basic Probability and Statistics** (e.g. CS 109 or equivalent)
    You should know the basics of probabilities, gaussian distributions, mean, standard deviation, etc.

- **Foundations of Machine Learning** (e.g. CS221, CS229, CS230, or CS124)
    We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. If you already have basic machine learning and/or deep learning knowledge, the course will be easier; however it is possible to take CS224n without it. There are many introductions to ML, in webpage, book, and video form. One approachable introduction is Hal Daumé’s in-progress A Course in Machine Learning. Reading the first 5 chapters of that book would be good background. Knowing the first 7 chapters would be even better! ### Reference Texts

The following texts are useful, but none are required. All of them can be read free online.

- Dan Jurafsky and James H. Martin. [Speech and Language Processing (3rd ed. draft)](https://web.stanford.edu/~jurafsky/slp3/)
- Jacob Eisenstein. [Natural Language Processing](https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf)
- Yoav Goldberg. [A Primer on Neural Network Models for Natural Language Processing](http://u.cs.biu.ac.il/~yogo/nnlp.pdf)
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. [Deep Learning](http://www.deeplearningbook.org/)
- Delip Rao and Brian McMahan. [Natural Language Processing with PyTorch](http://library.stanford.edu/sfx?genre=book&atitle=&title=Natural%20language%20processing%20with%20PyTorch%20:%20build%20intelligent%20language%20applications%20using%20deep%20learning%20/&isbn=9781491978207&volume=&issue=&date=20190101&aulast=Rao,%20Delip,,%20author.&spage=&pages=&sid=EBSCO:VLeBooks:edsvle.AH35866319) (requires Stanford login).
- Lewis Tunstall, Leandro von Werra, and Thomas Wolf. [Natural Language Processing with Transformers](https://transformersbook.com/)

If you have no background in neural networks but would like to take the course anyway, you might well find one of these books helpful to give you more background:

- Michael A. Nielsen. [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com)
- Eugene Charniak. [Introduction to Deep Learning](https://mitpress.mit.edu/books/introduction-deep-learning)

CS 224N: Natural Language Processing with Deep Learning

Natural Language Processing

Image Classification

BERT

Machine learning

Network Architectures

Optimizers

Segmentation and Detection

Deep Reinforcement Learning

Optimization

Sequential Data

Generative adversarial network (GAN)

CSE 490 G1 / 599 G1 Introduction to Deep Learning University of Washington Autumn 2019 A survey course on neural network implementation and applications, including image processing, classification, detection, and segmentation. The course also covers semantic understanding, translation, and question-answering applications. It's ideal for those with a background in Machine Learning, Neural Networks, Optimization, and CNNs. A survey class of neural network implementation and applications. Topics include: optimization - stochastic gradient descent, adaptive and 2nd order methods, normalization; convolutional neural networks - image processing, classification, detection, segmentation; recurrent neural networks - semantic understanding, translation, question-answering; cross-domain applications - image captioning, vision and language.   

CSE 490 G1 / 599 G1 Introduction to Deep Learning

Human Vision

Color Spaces

Transforms

Image Coordinates

Resizing

Filters

Convolution

Edges and Features

Harris Corner Detector

Feature Matching

RANSAC (Random Sample Consensus)

HOG (Histogram of Oriented Gradients)

SIFT (Scale-invariant Feature Transform)

Optical Flow

Depth Perception

Stereo

Semantic Segmentation

Object Detection

Instance Segmentation

Vision and Language

CSE 455 Computer Vision University of Washington Winter 2022 A general introduction to computer vision, this course covers traditional image processing techniques and newer, machine-learning based approaches. It discusses topics like filtering, edge detection, stereo, flow, and neural network architectures. This class is a general introduction to computer vision. It covers standard techniques in image processing like filtering, edge detection, stereo, flow, etc. (old-school vision), as well as newer, machine-learning based computer vision.   

CSE 455 Computer Vision

Connectionist Machines

McCullough and Pitt model

Hebb’s learning rule

Rosenblatt’s perceptron

Universal Approximator

Perceptron learning rule

Empirical risk minimization

Gradient descent

Back-propagation

Momentum

Nestorov

Convergence

Learning Rates

Optimization Algorithms

RMSProp

Stochastic gradient descent (SGD)

Acceleration

Overfitting

Regularization (mathematics)

Translation Invariance

Cascade Correlation Filters

Bidirectional RNNs

Sequence Prediction

Connectionist Temporal Classification (CTC)

Representations

Hopfield Networks

Boltzmann Machines

Normalizing Flows

Variational autoencoder (VAE)

Reinforcement learning (RL)

Multi-layer Perceptron

AdaGrad

11-785 Introduction to Deep Learning Carnegie Mellon University Spring 2020 This course provides a comprehensive introduction to deep learning, starting from foundational concepts and moving towards complex topics such as sequence-to-sequence models. Students gain hands-on experience with PyTorch and can fine-tune models through practical assignments. A basic understanding of calculus, linear algebra, and Python programming is required. “Deep Learning” systems, typified by deep neural networks, are increasingly taking over all AI tasks, ranging from language understanding, and speech and image recognition, to machine translation, planning, and even game playing and autonomous driving. As a result, expertise in deep learning is fast changing from an esoteric desirable to a mandatory prerequisite in many advanced academic settings, and a large advantage in the industrial job market.

In this course we will learn about the basics of deep neural networks, and their applications to various AI tasks. By the end of the course, it is expected that students will have significant familiarity with the subject, and be able to apply Deep Learning to a variety of tasks. They will also be positioned to understand much of the current literature on the topic and extend their knowledge through further study.

If you are only interested in the lectures, you can watch them on the YouTube channel listed below.

### Course description from student point of view

The course is well rounded in terms of concepts. It helps us understand the fundamentals of Deep Learning. The course starts off gradually with MLPs and it progresses into the more complicated concepts such as attention and sequence-to-sequence models. We get a complete hands on with PyTorch which is very important to implement Deep Learning models. As a student, you will learn the tools required for building Deep Learning models. The homeworks usually have 2 components which is Autolab and Kaggle. The Kaggle components allow us to explore multiple architectures and understand how to fine-tune and continuously improve models. The task for all the homeworks were similar and it was interesting to learn how the same task can be solved using multiple Deep Learning approaches. Overall, at the end of this course you will be confident enough to build and tune Deep Learning models.  1. We will be using one of several toolkits (the primary toolkit for recitations/instruction is PyTorch). The toolkits are largely programmed in Python. You will need to be able to program in at least one of these languages. Alternately, you will be responsible for finding and learning a toolkit that requires programming in a language you are comfortable with,
1. You will need familiarity with basic calculus (differentiation, chain rule), linear algebra and basic probability. ### Course description from student point of view

The course is well rounded in terms of concepts. It helps us understand the fundamentals of Deep Learning. The course starts off gradually with MLPs and it progresses into the more complicated concepts such as attention and sequence-to-sequence models. We get a complete hands on with PyTorch which is very important to implement Deep Learning models. As a student, you will learn the tools required for building Deep Learning models. The homeworks usually have 2 components which is Autolab and Kaggle. The Kaggle components allow us to explore multiple architectures and understand how to fine-tune and continuously improve models. The task for all the homeworks were similar and it was interesting to learn how the same task can be solved using multiple Deep Learning approaches. Overall, at the end of this course you will be confident enough to build and tune Deep Learning models.

11-785 Introduction to Deep Learning

Decision Tree Learning

Margins

Naive Bayes

Maximizing Conditional Likelihood

Kernelizing Algorithms

Primal and Dual Forms

Finite Hypothesis Classes

k-fold cross-validation

Objective-Based Clustering

Principal Component Analysis (PCA)

Semi-Supervised Learning

Learning Linear Separators

Maximum Likelihood Estimation (MLE)

Bag of Words

Generalization and Overfitting

Model Selection and Regularization

Support Vector Machine (SVM)

Markov Decision Process (MDP)

Minimizing Squared Error

Deep Networks

Adaboost

Learning Representations

Active Learning

Co-training

Q-learning

Conditional Independence

Kernelizing Perceptron

Kernelizing SVM

VC Dimension Based Bounds

Linear regression

Boosting

Hierarchical Clustering

Interactive Learning

Transductive SVM

Value iteration

Bayes' theorem

Perceptron

Maximum A Posteriori (MAP)

Logistic Regression

Kernel (operating system)

Geometric Margins

Sample Complexity

Structural Risk Minimization

Maximizing Data Likelihood

Unsupervised learning

Dimensionality Reduction

Sampling Bias

10-401 Introduction to Machine Learning Carnegie Mellon University Spring 2018 A comprehensive exploration of machine learning theories and practical algorithms. Covers a broad spectrum of topics like decision tree learning, neural networks, statistical learning, and reinforcement learning. Encourages hands-on learning via programming assignments.  Machine Learning is concerned with computer programs that automatically improve their performance through experience (e.g., programs that learn to recognize human faces, recommend music and movies, and drive autonomous robots). This course covers the theory and practical algorithms for machine learning from a variety of perspectives. We cover topics such as decision tree learning, Support Vector Machines, neural networks, boosting, statistical learning methods, unsupervised learning, active leaerning, and reinforcement learning. Short programming assignments include hands-on experiments with various learning algorithms.   - Machine Learning, Tom Mitchell. (optional)
- Pattern Recognition and Machine Learning, Christopher Bishop. (optional)
- Machine Learning: A Probabilistic Perspective, Kevin P. Murphy, [available online](https://ebookcentral.proquest.com/lib/cm/detail.action?docID=3339490), (optional)

10-401 Introduction to Machine Learning

Machine Learning

Image Filtering

Image Pyramids

Frequency Domain

Hough Transform

Detecting Corners

Feature Detectors and Descriptors

2D Transformations

Image Homographies

Geometric Camera Models

Two-View Geometry

Alignment and Tracking

Radiometry and Reflectance

Photometric Stereo

Digital Photography

16-385 Computer Vision Carnegie Mellon University Spring 2022 This course gives an expansive introduction to computer vision, focusing on image processing, recognition, geometry-based and physics-based vision, and video analysis. Students will gain practical experience solving real-life vision problems. It requires a good understanding of linear algebra, calculus, and programming. This course provides a comprehensive introduction to computer vision. Major topics include image processing, detection and recognition, geometry-based and physics-based vision and video analysis. Students will learn basic concepts of computer vision as well as hands on experience to solve real-life vision problems.  This course requires familiarity with linear algebra, calculus, basic probability, as well as programming. In particular, the following courses serve as prerequisite:

- "Mathematical Foundations of Electrical Engineering" (18-202) and "Principles of Imperative Computation" (15-122)

OR

- "Matrix Algebra with Applications" (21-240) and "Matrices and Linear Transformations" (21-241) and "Calculus in Three Dimensions" (21-259) and "Principles of Imperative Computation" (15-122) Readings will be assigned from the following textbook (available online for free):

- **Computer Vision**: Algorithms and Applications, by Richard Szeliski.
Additional readings will be assigned from relevant papers. Readings will be posted on the website.

The following textbooks can also be useful references for different parts of the class, but are not required:

- Multiple View Geometry in Computer Vision, by Richard Hartley and Andrew Zisserman.
- Computer Vision: A Modern Approach, by David Forsyth and Jean Ponce.
- Digital Image Processing, by Rafael Gonzalez and Richard Woods.

16-385 Computer Vision

Supervised learning

Classification

Training/Validation/Testing

MNIST

Loss function

Automatic differentiation

Matrix representation of Neural Networks

Graphics Processor (GPU)

TensorFlow

Multilayer perceptron

Activation Functions

Word Embeddings

Feedforward neural network

Gated recurrent unit (GRU)

Machine Translation

Scaling Deep Learning Systems

Interpretation of Neural Networks

Deepfake

Deep Q-learning

Policy Gradient Methods

Actor-Critic Methods

CSCI 1470/2470 Deep Learning Brown University Spring 2022 Brown University's Deep Learning course acquaints students with the transformative capabilities of deep neural networks in computer vision, NLP, and reinforcement learning. Using the TensorFlow framework, topics like CNNs, RNNs, deepfakes, and reinforcement learning are addressed, with an emphasis on ethical applications and potential societal impacts. Over the past few years, Deep Learning has become a popular area, with deep neural network methods obtaining state-of-the-art results on applications in computer vision (Self-Driving Cars), natural language processing (Google Translate), and reinforcement learning (AlphaGo). These technologies are having transformative effects on our society, including some undesirable ones (e.g. deep fakes).

This course is there to give students a practical understanding of how Deep Learning works, how to implement neural networks, and how to apply them ethically. We introduce students to the core concepts of deep neural networks and survey the techniques used to model complex processes within the contexts of computer vision and natural language processing.

Throughout the course, we emphasize and require students to think critically about potential ethical pitfalls that can result from mis-application of these powerful models. The course is taught using the Tensorflow deep learning framework.
 By the end of this course, you will be able to:

- Learn about the fundamental algorithms that underly all modern deep learning models.
- Implement different types of deep learning models in Tensorflow.
- Think critically about using a deep learning model for a task and its potential societal impact.
- Collaborate with classmates on a team project to apply deep learning models to task of your
choice.
- Communicate your findings (both positive and negative results are encouraged) through pre-
sentations. - A basic programming course: (CSCI 0150, 0170 or 0190)
- A linear algebra course: (CSCI 0530, MATH 0520 or 0540)
- A stats / probability course: (CSCI 0220, 1450, 0450, MATH 1610, APMA 1650 or 1655)

Exceptions may be possible for those missing one of these prerequisites if (a) the student has taken
another course which covers similar material, or if (b) the student will be concurrently taking the
prerequisite. If either of these situations applies to you, use the “Request Override” feature in
Courses@Brown to request an override code (and explain why you believe your situation merits
one). ### Textbook

None required. Students are encouraged to refer to the following textbook, which is available online:

- [Deep Learning](https://www.deeplearningbook.org/), by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

CSCI 1470/2470 Deep Learning

Convolutional neural network

Convolutional neural network (CNN)

8 courses cover this concept