Q-learning

PyTorch

Exploration / Exploitation

Imitation learning

Tabular MDP planning

Policy Search

Tabular RL Policy Evaluation

Batch Reinforcement Learning

Reinforcement learning (RL)

Function approximation

Value Alignment

CS 234: Reinforcement Learning Stanford University Winter 2023 This course offers a solid introduction to the field of reinforcement learning (RL), covering challenges, approaches, and deep RL. Prerequisites include Python proficiency and foundations of machine learning. Students will be able to implement RL algorithms and evaluate them. To realize the dreams and impact of AI requires autonomous systems that learn to make good decisions. Reinforcement learning is one powerful paradigm for doing so, and it is relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare. This class will provide a solid introduction to the field of reinforcement learning and students will learn about the core challenges and approaches, including generalization and exploration. Through a combination of lectures, and written and coding assignments, students will become well versed in key ideas and techniques for RL. Assignments will include the basics of reinforcement learning as well as deep reinforcement learning — an extremely promising new area that combines deep learning techniques with reinforcement learning. By the end of the class students should be able to:

- Define the key features of reinforcement learning that distinguishes it from AI and non-interactive machine learning (as assessed by the exam).
- Given an application problem (e.g. from computer vision, robotics, etc), decide if it should be formulated as a RL problem; if yes be able to define it formally (in terms of the state space, action space, dynamics and reward model), state what algorithm (from class) is best suited for addressing it and justify your answer (as assessed by the exam).
- Implement in code common RL algorithms (as assessed by the assignments).
- Describe (list and define) multiple criteria for analyzing RL algorithms and evaluate algorithms on these metrics: e.g. regret, sample complexity, computational complexity, empirical performance, convergence, etc (as assessed by assignments and the exam).
- Describe the exploration vs exploitation challenge and compare and contrast at least two approaches for addressing this challenge (in terms of performance, scalability, complexity of implementation, and theoretical guarantees) (as assessed by an assignment and the exam). - Proficiency in Python  
     All class assignments will be in Python. There is a tutorial [here](http://cs231n.github.io/python-numpy-tutorial/) for those who aren't as familiar with Python. If you have a lot of programming experience but in a different language (e.g. C/ C++/ Matlab/ Javascript) you will probably be fine.
- College Calculus, Linear Algebra (e.g. MATH 51, CME 100)  
     You should be comfortable taking derivatives and understanding matrix vector operations and notation.
- Basic Probability and Statistics (e.g. CS 109 or other stats course)  
     You should know basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
- Foundations of Machine Learning  
     We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. Either CS 221 or CS 229 cover this background. Some optimization tricks will be more intuitive with some knowledge of convex optimization. There is no official textbook for the class but a number of the supporting readings will come from:

- Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition. This is available for free [here](http://incompleteideas.net/book/the-book-2nd.html) and references will refer to the final pdf version available [here](http://incompleteideas.net/book/RLbook2018.pdf).

Some other additional references that may be useful are listed below:

- Reinforcement Learning: State-of-the-Art, Marco Wiering and Martijn van Otterlo, Eds. \[[link](https://link.springer.com/book/10.1007%2F978-3-642-27645-3)\]
- Artificial Intelligence: A Modern Approach, Stuart J. Russell and Peter Norvig.\[[link](http://aima.cs.berkeley.edu/)\]
- Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville. \[[link](http://www.deeplearningbook.org/)\]
- David Silver's course on Reinforcement Learning \[[link](http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching.html)\]

CS 234: Reinforcement Learning

Artificial Intelligence

Histories of AI

Societal impacts of AI

Vector (mathematics and physics)

Dot product

Geometric interpretations

Taking gradients

Discrete random variables

Probability distributions

Mean

Variance

Marginal distributions

Conditional distribution

Big-O notation

Computational complexity

Recurrence relation

Dynamic programming

Continuous optimization

Objective functions

Gradient descent

Machine learning

Loss minimization

Hinge loss

Equitable performance

Design and organization of features

Computation graphs

Deep learning models composition

Cross-validation (statistics)

Search problem

Breadth-first search (BFS)

Uniform cost search (UCS)

UCS heuristics

Markov Decision Process (MDP)

Transportation problem

Discounting factor

Model-free Monte Carlo

Game theory

Minimax algorithm

Temporal difference (TD) learning

Variable-based models

Backtracking search

Gibbs sampling

Bayesian network

Laplace smoothing

Logic

Entailment

Satisfiability

Soundness

Modus ponens

First-order logic

Unification (computer science)

Python

Linear regression

Linear classification

Stochastic gradient descent (SGD)

Non-linear functions

Neural network

Backpropagation

Generalization

K-means clustering

Exhaustive search

Depth-first search (DFS)

UCS correctness

Relaxed search problems

Dice game

Policy evaluation

Value iteration

Model-based Monte Carlo

State–action–reward–state–action (SARSA)

Epsilon-greedy exploration

Deep Reinforcement Learning

Halving game

Alpha-beta pruning

Nash equilibrium

Factor graphs

AC-3 algorithm

Markov random field

Hidden Markov Model (HMM)

Maximum Likelihood Estimation (MLE)

Propositional calculus (Propositional logic)

Contradiction

Model checking

Completeness

Horn clauses

Substitution (logic)

Skolem functions

CS 221 Artificial Intelligence: Principles and Techniques Stanford University Autumn 2022-2023 Stanford's CS 221 course teaches foundational principles and practical implementation of AI systems. It covers machine learning, game playing, constraint satisfaction, graphical models, and logic. A rigorous course requiring solid foundational skills in programming, math, and probability. The goal of artificial intelligence (AI) is to tackle complex real-world problems with rigorous mathematical tools. In this course, you will learn the foundational principles and practice implementing various AI systems. Specific topics include machine learning, search, Markov decision processes, game playing, constraint satisfaction, graphical models, and logic.  This course is fast-paced and covers a lot of ground, so it is important that you have a solid foundation in a number of areas. Here are the basic skills that you need and the classes that teach those skills:

- Programming (ideally Python): [CS 106A](http://www.stanford.edu/class/cs106a/), [CS 106B](http://www.stanford.edu/class/cs106b/), [CS 107](http://www.stanford.edu/class/cs107/)
- Discrete math, mathematical rigor: [CS 103](http://www.stanford.edu/class/cs103/)
- Probability: [CS 109](http://www.stanford.edu/class/cs109/)
- Linear algebra: [Math 51](https://web.stanford.edu/class/math51/textbook.html)

It is less important that you know particular things (e.g., we don't use eigenvectors in this course even though that's a pillar of any linear algebra course), and more important that you've done enough related things that you feel at ease with it. While it is possible to fill in the gaps, this course does move quickly, and ideally you want to be focusing your energy on learning AI rather than catching up on prerequisites. We have made a few [prerequisite modules](https://stanford-cs221.github.io/autumn2022/modules) that you can review to refresh your memory, and the first homework (foundations) will allow you to also get some practice on these basics. ### Further Reading

There are no required textbooks for this class, and you should be able to learn everything from the lecture notes and homeworks. However, if you would like to pursue more advanced topics or get another perspective on the same material, here are some great resources:

- [Russell and Norvig. Artificial Intelligence: A Modern Approach](http://aima.cs.berkeley.edu/). A comprehensive reference for all the AI topics that we will cover.
- [Koller and Friedman. Probabilistic Graphical Models](http://mitpress.mit.edu/books/probabilistic-graphical-models). Covers factor graphs and Bayesian networks (this is the textbook for CS228).
- [Sutton and Barto. Reinforcement Learning: An Introduction](https://mitpress.mit.edu/books/reinforcement-learning). Covers Markov decision processes and reinforcement learning (free online).
- [Hastie, Tibshirani, and Friedman. The Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn/). Covers machine learning from a rigorous statistical perspective (free online).
- [Tsang. Foundations of Constraint Satisfaction](http://www.bracil.net/edward/fcs.html). Covers constraint satisfaction problems (free online).

Note that some of these books use different notation and terminology from this course, so it may take some effort to make the appropriate connections.

CS 221 Artificial Intelligence: Principles and Techniques

Contractions

Asynchronous value iteration

Linear–quadratic regulator (LQR)

Differential dynamic programming (DDP)

Quadruped locomotion

Partially observable Markov decision process (POMDP)

Bandits

Separation Principle

Dynamics Modeling

Kalman Filtering

Policy Gradient

TD-Gammon

Reward Shaping

Linear Programming Approach

Inverse reinforcement learning

Model Predictive Control

Simultaneous Localization and Mapping (SLAM)

Linearly-solvable Markov decision problems

Learning to walk

Policy Iteration

CS 294-40: Learning for robotics and control UC Berkeley Fall 2008 This advanced course focuses on the applications of machine learning in the robotics and control field. It covers a wide range of topics including Markov Decision Processes, control theories, estimation methodologies, and robotics principles. Recommended for graduate students. This is an advanced course in learning for robotics and control. The goal of this course is to help the audience with their research in learning for robotics and control or related topics. A tentative list of topics includes:

- Markov decision processes: value iteration, policy iteration, linear programming, Q learning, TD, value function approximation, inverse reinforcement learning
- Control: linear quadratic regulator, differential dynamic programming, receding horizon / model predictive control
- Estimation: (extended) Kalman filters, particle filters, SLAM
- Robotics: basic principles of various robots, sensors, microcontrollers
- Exploration/Exploitation: bandits, no-regret, e^3  Familiarity with mathematical proofs, machine learning, artificial intelligence, optimization, probability, algorithms, linear algebra; ability to implement algorithmic ideas in code (C/C++ and matlab).

Graduate students only (consent of instructor required for undergraduate students, please talk to me after first lecture and hand me summary of relevant classes/experience so I can decide whether to make an exception). 

CS 294-40: Learning for robotics and control

Robotics

Decision Tree Learning

Margins

Naive Bayes

Maximizing Conditional Likelihood

Kernelizing Algorithms

Primal and Dual Forms

Finite Hypothesis Classes

k-fold cross-validation

Convolutional neural network (CNN)

Objective-Based Clustering

Principal Component Analysis (PCA)

Semi-Supervised Learning

Learning Linear Separators

Bag of Words

Computer Vision

Generalization and Overfitting

Model Selection and Regularization

Support Vector Machine (SVM)

Minimizing Squared Error

Deep Networks

Adaboost

Learning Representations

Active Learning

Co-training

Overfitting

Conditional Independence

Kernelizing Perceptron

Kernelizing SVM

VC Dimension Based Bounds

Boosting

Hierarchical Clustering

Interactive Learning

Transductive SVM

Bayes' theorem

Perceptron

Maximum A Posteriori (MAP)

Logistic Regression

Kernel (operating system)

Geometric Margins

Sample Complexity

Structural Risk Minimization

Maximizing Data Likelihood

Unsupervised learning

Dimensionality Reduction

Sampling Bias

Convolution

10-401 Introduction to Machine Learning Carnegie Mellon University Spring 2018 A comprehensive exploration of machine learning theories and practical algorithms. Covers a broad spectrum of topics like decision tree learning, neural networks, statistical learning, and reinforcement learning. Encourages hands-on learning via programming assignments.  Machine Learning is concerned with computer programs that automatically improve their performance through experience (e.g., programs that learn to recognize human faces, recommend music and movies, and drive autonomous robots). This course covers the theory and practical algorithms for machine learning from a variety of perspectives. We cover topics such as decision tree learning, Support Vector Machines, neural networks, boosting, statistical learning methods, unsupervised learning, active leaerning, and reinforcement learning. Short programming assignments include hands-on experiments with various learning algorithms.   - Machine Learning, Tom Mitchell. (optional)
- Pattern Recognition and Machine Learning, Christopher Bishop. (optional)
- Machine Learning: A Probabilistic Perspective, Kevin P. Murphy, [available online](https://ebookcentral.proquest.com/lib/cm/detail.action?docID=3339490), (optional)

10-401 Introduction to Machine Learning

Machine Learning

Q-learning

4 courses cover this concept

CS 234: Reinforcement Learning

CS 221 Artificial Intelligence: Principles and Techniques

CS 294-40: Learning for robotics and control

10-401 Introduction to Machine Learning