Self-Attention

Word Vectors

Word Window Classification

Language Models

Backpropagation

Neural network

Dependency Parsing

Recurrent neural network (RNN)

Sequence-to-sequence (Seq2Seq)

Subword Models

Transformer (machine learning model)

Natural Language Generation

Pretraining

Reinforcement Learning from Human Feedback (RLHF)

Question Answering (QA)

Convolutional neural network (CNN)

Tree Recursive Neural Networks

Constituency Parsing

Code Generation

Multimodal Deep Learning

Coreference Resolution

CS 224N: Natural Language Processing with Deep Learning Stanford University Winter 2023 CS 224N provides an in-depth introduction to neural networks for NLP, focusing on end-to-end neural models. The course covers topics such as word vectors, recurrent neural networks, and transformer models, among others. Natural language processing (NLP) is a crucial part of artificial intelligence (AI), modeling how people share information. In recent years, deep learning approaches have obtained very high performance on many NLP tasks. In this course, students gain a thorough introduction to cutting-edge neural networks for NLP. ## What is this course about?

Natural language processing (NLP) or computational linguistics is one of the most important technologies of the information age. Applications of NLP are everywhere because people communicate almost everything in language: web search, advertising, emails, customer service, language translation, virtual agents, medical reports, politics, etc. In the last decade, deep learning (or neural network) approaches have obtained very high performance across many different NLP tasks, using single end-to-end neural models that do not require traditional, task-specific feature engineering. In this course, students will gain a thorough introduction to cutting-edge research in Deep Learning for NLP. Through lectures, assignments and a final project, students will learn the necessary skills to design, implement, and understand their own neural network models, using the Pytorch framework.

>  “Take it. CS221 taught me algorithms. CS229 taught me math. CS224N taught me how to write machine learning models.” – A CS224N student on Carta - **Proficiency in Python**
    All class assignments will be in Python (using NumPy and PyTorch). If you need to remind yourself of Python, or you're not very familiar with NumPy, you can come to the Python review session in week 1 (listed in the schedule). If you have a lot of programming experience but in a different language (e.g. C/C++/Matlab/Java/Javascript), you will probably be fine.
- **College Calculus, Linear Algebra** (e.g. MATH 51, CME 100)
    You should be comfortable taking (multivariable) derivatives and understanding matrix/vector notation and operations.

- **Basic Probability and Statistics** (e.g. CS 109 or equivalent)
    You should know the basics of probabilities, gaussian distributions, mean, standard deviation, etc.

- **Foundations of Machine Learning** (e.g. CS221, CS229, CS230, or CS124)
    We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. If you already have basic machine learning and/or deep learning knowledge, the course will be easier; however it is possible to take CS224n without it. There are many introductions to ML, in webpage, book, and video form. One approachable introduction is Hal Daumé’s in-progress A Course in Machine Learning. Reading the first 5 chapters of that book would be good background. Knowing the first 7 chapters would be even better! ### Reference Texts

The following texts are useful, but none are required. All of them can be read free online.

- Dan Jurafsky and James H. Martin. [Speech and Language Processing (3rd ed. draft)](https://web.stanford.edu/~jurafsky/slp3/)
- Jacob Eisenstein. [Natural Language Processing](https://github.com/jacobeisenstein/gt-nlp-class/blob/master/notes/eisenstein-nlp-notes.pdf)
- Yoav Goldberg. [A Primer on Neural Network Models for Natural Language Processing](http://u.cs.biu.ac.il/~yogo/nnlp.pdf)
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. [Deep Learning](http://www.deeplearningbook.org/)
- Delip Rao and Brian McMahan. [Natural Language Processing with PyTorch](http://library.stanford.edu/sfx?genre=book&atitle=&title=Natural%20language%20processing%20with%20PyTorch%20:%20build%20intelligent%20language%20applications%20using%20deep%20learning%20/&isbn=9781491978207&volume=&issue=&date=20190101&aulast=Rao,%20Delip,,%20author.&spage=&pages=&sid=EBSCO:VLeBooks:edsvle.AH35866319) (requires Stanford login).
- Lewis Tunstall, Leandro von Werra, and Thomas Wolf. [Natural Language Processing with Transformers](https://transformersbook.com/)

If you have no background in neural networks but would like to take the course anyway, you might well find one of these books helpful to give you more background:

- Michael A. Nielsen. [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com)
- Eugene Charniak. [Introduction to Deep Learning](https://mitpress.mit.edu/books/introduction-deep-learning)

CS 224N: Natural Language Processing with Deep Learning

Natural Language Processing

K-Nearest Neighbors

Linear Classifiers

Stochastic gradient descent (SGD)

Momentum

AdaGrad

Adam

Learning Rate Schedules

Support Vector Machine (SVM)

Softmax Loss

Multi-layer Perceptron

Higher-level representations

Image Features

Pooling

Batch Normalization

Transfer learning

AlexNet, VGG, GoogLeNet, ResNet

Activation Functions

Data Processing

Weight initialization

Hyperparameter Tuning

Data Augmentation

Feature visualization and inversion

Adversarial Examples

DeepDream and Style Transfer

Single-stage detectors

Two-stage detectors

Semantic/Instance/Panoptic segmentation

Long Short-Term Memory (LSTM)

Gated recurrent unit (GRU)

Language Modeling

Image Captioning

Object Detection

Video classification

3D CNNs

Two-stream networks

Multimodal Video Understanding

Supervised learning

Unsupervised learning

Pixel RNN, Pixel CNN

Variational autoencoder (VAE)

Generative adversarial network (GAN)

Pretext Tasks

Contrastive Learning

Multisensory Supervision

Optical Flow

Stereo Vision

3D Shape Representations

Shape Reconstruction

Neural Implicit Representations

Convolution

CS231n: Deep Learning for Computer Vision Stanford University Spring 2022 This is a deep-dive into the details of deep learning architectures for visual recognition tasks. The course provides students with the ability to implement, train their own neural networks and understand state-of-the-art computer vision research. It requires Python proficiency and familiarity with calculus, linear algebra, probability, and statistics. Computer Vision has become ubiquitous in our society, with applications in search, image understanding, apps, mapping, medicine, drones, and self-driving cars. Core to many of these applications are visual recognition tasks such as image classification, localization and detection. Recent developments in neural network (aka “deep learning”) approaches have greatly advanced the performance of these state-of-the-art visual recognition systems. This course is a deep dive into the details of deep learning architectures with a focus on learning end-to-end models for these tasks, particularly image classification. During the 10-week course, students will learn to implement and train their own neural networks and gain a detailed understanding of cutting-edge research in computer vision. Additionally, the final assignment will give them the opportunity to train and apply multi-million parameter networks on real-world vision problems of their choice. Through multiple hands-on assignments and the final course project, students will acquire the toolset for setting up deep learning tasks and practical engineering tricks for training and fine-tuning deep neural networks.  - Proficiency in Python  
    All class assignments will be in Python (and use numpy) (we provide a tutorial [here](http://cs231n.github.io/python-numpy-tutorial/) for those who aren't as familiar with Python). If you have a lot of programming experience but in a different language (e.g. C/C++/Matlab/Javascript) you will probably be fine.
- College Calculus, Linear Algebra (e.g. MATH 19 or 41, MATH 51)  
     You should be comfortable taking derivatives and understanding matrix vector operations and notation.
- Basic Probability and Statistics (e.g. CS 109 or other stats course)  
    You should know basics of probabilities, gaussian distributions, mean, standard deviation, etc. 

CS231n: Deep Learning for Computer Vision

Deep Learning

Computer Vision

Self-attention

Self-Attention

2 courses cover this concept

CS 224N: Natural Language Processing with Deep Learning

CS231n: Deep Learning for Computer Vision