Policy Search

PyTorch

Exploration / Exploitation

Q-learning

Imitation learning

Tabular MDP planning

Tabular RL Policy Evaluation

Batch Reinforcement Learning

Reinforcement learning (RL)

Function approximation

Value Alignment

CS 234: Reinforcement Learning Stanford University Winter 2023 This course offers a solid introduction to the field of reinforcement learning (RL), covering challenges, approaches, and deep RL. Prerequisites include Python proficiency and foundations of machine learning. Students will be able to implement RL algorithms and evaluate them. To realize the dreams and impact of AI requires autonomous systems that learn to make good decisions. Reinforcement learning is one powerful paradigm for doing so, and it is relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare. This class will provide a solid introduction to the field of reinforcement learning and students will learn about the core challenges and approaches, including generalization and exploration. Through a combination of lectures, and written and coding assignments, students will become well versed in key ideas and techniques for RL. Assignments will include the basics of reinforcement learning as well as deep reinforcement learning — an extremely promising new area that combines deep learning techniques with reinforcement learning. By the end of the class students should be able to:

- Define the key features of reinforcement learning that distinguishes it from AI and non-interactive machine learning (as assessed by the exam).
- Given an application problem (e.g. from computer vision, robotics, etc), decide if it should be formulated as a RL problem; if yes be able to define it formally (in terms of the state space, action space, dynamics and reward model), state what algorithm (from class) is best suited for addressing it and justify your answer (as assessed by the exam).
- Implement in code common RL algorithms (as assessed by the assignments).
- Describe (list and define) multiple criteria for analyzing RL algorithms and evaluate algorithms on these metrics: e.g. regret, sample complexity, computational complexity, empirical performance, convergence, etc (as assessed by assignments and the exam).
- Describe the exploration vs exploitation challenge and compare and contrast at least two approaches for addressing this challenge (in terms of performance, scalability, complexity of implementation, and theoretical guarantees) (as assessed by an assignment and the exam). - Proficiency in Python  
     All class assignments will be in Python. There is a tutorial [here](http://cs231n.github.io/python-numpy-tutorial/) for those who aren't as familiar with Python. If you have a lot of programming experience but in a different language (e.g. C/ C++/ Matlab/ Javascript) you will probably be fine.
- College Calculus, Linear Algebra (e.g. MATH 51, CME 100)  
     You should be comfortable taking derivatives and understanding matrix vector operations and notation.
- Basic Probability and Statistics (e.g. CS 109 or other stats course)  
     You should know basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
- Foundations of Machine Learning  
     We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. Either CS 221 or CS 229 cover this background. Some optimization tricks will be more intuitive with some knowledge of convex optimization. There is no official textbook for the class but a number of the supporting readings will come from:

- Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition. This is available for free [here](http://incompleteideas.net/book/the-book-2nd.html) and references will refer to the final pdf version available [here](http://incompleteideas.net/book/RLbook2018.pdf).

Some other additional references that may be useful are listed below:

- Reinforcement Learning: State-of-the-Art, Marco Wiering and Martijn van Otterlo, Eds. \[[link](https://link.springer.com/book/10.1007%2F978-3-642-27645-3)\]
- Artificial Intelligence: A Modern Approach, Stuart J. Russell and Peter Norvig.\[[link](http://aima.cs.berkeley.edu/)\]
- Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville. \[[link](http://www.deeplearningbook.org/)\]
- David Silver's course on Reinforcement Learning \[[link](http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching.html)\]

CS 234: Reinforcement Learning

Artificial Intelligence

Probability

Structure Learning

Beliefs

Multiagent Reasoning

Inference

Exact Solutions

Actor Critic

Offline Belief State Planning

Parameter Learning

Online Planning

Model-Based and Model-Free Reinforcement Learning

Controller Abstractions

Simple Decisions

Policy Gradient Estimation and Optimization

Exact Belief State Planning

Policy Validation, Implications, and Summary

Representations

Bayesian Networks Representation and Inference

Approximate Value Functions

Online Belief State Planning

AA 228 / CS 238 Decision Making under Uncertainty Stanford University Winter 2023 The course introduces decision making under uncertainty from a computational perspective, covering dynamic programming, reinforcement learning, and more. Prerequisites include basic probability and fluency in a high-level programming language. This course introduces decision making under uncertainty from a computational perspective and provides an overview of the necessary tools for building autonomous and decision-support systems. Following an introduction to probabilistic models and decision theory, the course will cover computational methods for solving decision problems with stochastic dynamics, model uncertainty, and imperfect state information. Topics include Bayesian networks, influence diagrams, dynamic programming, reinforcement learning, and partially observable Markov decision processes. Applications cover air traffic control, aviation surveillance systems, autonomous vehicles, and robotic planetary exploration. - You will gain a broad fundamental understanding of the mathematical models and solution methods for decision making (exercises, three quizzes).
- You will be able to implement and extend key algorithms for learning and decision making (two programming projects).
- You will be able to identify an application of the theory in this course and formulate it mathematically (proposal).
- You will gain a deep understanding of an area of particular interest and apply it to a problem (final project).
- You will be able to critique approaches to solving decision problems (peer review). basic probability and fluency in a high-level programming language. This year we will be using the textbook titled *Algorithms for Decision Making*, MIT Press, 2022. Printed copies are available for purchase or you may download it for free as a PDF. Details are [here](https://algorithmsbook.com/decisionmaking).

The textbook serves as the official lecture notes.

AA 228 / CS 238 Decision Making under Uncertainty

Policy Search

2 courses cover this concept

CS 234: Reinforcement Learning

AA 228 / CS 238 Decision Making under Uncertainty