Reinforcement learning (RL)

Markov Decision Process (MDP)

Hidden Markov Model (HMM)

Neural network

Probability

Intelligent agent

Search

Game theory

Adversarial Search

Knowledge representation and reasoning

Logic

Uncertain Knowledge

Bayes' theorem

Bayesian network

Robot Motion Planning

Classical Planning

Probabilistic Planning

Supervised learning

Unsupervised learning

Natural Language Processing (NLP)

Robotics

Computer Vision

Philosophy of AI

Social and Ethical Issues in AI

Ethical Dilemmas in AI

CS1410 Artificial Intelligence Brown University Fall 2022 CS1410 at Brown University delves into the realm of Artificial Intelligence. Using the 3rd edition of "Artificial Intelligence, A Modern Approach" by Russell & Norvig, students explore intelligent agents, game theory, knowledge representation, logic, probabilistic learning, NLP, robotics, computer vision, and ethical implications of AI.   Prerequisites to CS1410 include (CSCI 0160, 0180, 0190, or 0200) and (CSCI 0220, 1450, 0450, APMA 1650 or 1655) or minimum score of WAIVE in 'Graduate Student PreReq'. ### Required Text

*Artificial Intelligence, A Modern Approach* Russell & Norvig, **3rd** Edition.

### On Lectures

The textbook contains everything you need to know.

Lectures contain everything you need to know.

Lecture notes **do not contain everything you need to
know**.

CS1410 Artificial Intelligence

Histories of AI

Societal impacts of AI

Vector (mathematics and physics)

Dot product

Geometric interpretations

Taking gradients

Discrete random variables

Probability distributions

Mean

Variance

Marginal distributions

Conditional distribution

Big-O notation

Computational complexity

Recurrence relation

Dynamic programming

Continuous optimization

Objective functions

Gradient descent

Python

Machine learning

Linear regression

Loss minimization

Linear classification

Hinge loss

Stochastic gradient descent (SGD)

Equitable performance

Non-linear functions

Design and organization of features

Computation graphs

Backpropagation

Deep learning models composition

Generalization

Cross-validation (statistics)

K-means clustering

Search problem

Exhaustive search

Breadth-first search (BFS)

Depth-first search (DFS)

Uniform cost search (UCS)

UCS correctness

UCS heuristics

Relaxed search problems

Dice game

Transportation problem

Policy evaluation

Discounting factor

Value iteration

Model-based Monte Carlo

Model-free Monte Carlo

State–action–reward–state–action (SARSA)

Q-learning

Epsilon-greedy exploration

Function approximation

Deep Reinforcement Learning

Halving game

Minimax algorithm

Alpha-beta pruning

Temporal difference (TD) learning

Nash equilibrium

Variable-based models

Factor graphs

Backtracking search

AC-3 algorithm

Gibbs sampling

Markov random field

Laplace smoothing

Maximum Likelihood Estimation (MLE)

Propositional calculus (Propositional logic)

Entailment

Contradiction

Satisfiability

Model checking

Soundness

Completeness

Modus ponens

Horn clauses

First-order logic

Substitution (logic)

Unification (computer science)

Skolem functions

CS 221 Artificial Intelligence: Principles and Techniques Stanford University Autumn 2022-2023 Stanford's CS 221 course teaches foundational principles and practical implementation of AI systems. It covers machine learning, game playing, constraint satisfaction, graphical models, and logic. A rigorous course requiring solid foundational skills in programming, math, and probability. The goal of artificial intelligence (AI) is to tackle complex real-world problems with rigorous mathematical tools. In this course, you will learn the foundational principles and practice implementing various AI systems. Specific topics include machine learning, search, Markov decision processes, game playing, constraint satisfaction, graphical models, and logic.  This course is fast-paced and covers a lot of ground, so it is important that you have a solid foundation in a number of areas. Here are the basic skills that you need and the classes that teach those skills:

- Programming (ideally Python): [CS 106A](http://www.stanford.edu/class/cs106a/), [CS 106B](http://www.stanford.edu/class/cs106b/), [CS 107](http://www.stanford.edu/class/cs107/)
- Discrete math, mathematical rigor: [CS 103](http://www.stanford.edu/class/cs103/)
- Probability: [CS 109](http://www.stanford.edu/class/cs109/)
- Linear algebra: [Math 51](https://web.stanford.edu/class/math51/textbook.html)

It is less important that you know particular things (e.g., we don't use eigenvectors in this course even though that's a pillar of any linear algebra course), and more important that you've done enough related things that you feel at ease with it. While it is possible to fill in the gaps, this course does move quickly, and ideally you want to be focusing your energy on learning AI rather than catching up on prerequisites. We have made a few [prerequisite modules](https://stanford-cs221.github.io/autumn2022/modules) that you can review to refresh your memory, and the first homework (foundations) will allow you to also get some practice on these basics. ### Further Reading

There are no required textbooks for this class, and you should be able to learn everything from the lecture notes and homeworks. However, if you would like to pursue more advanced topics or get another perspective on the same material, here are some great resources:

- [Russell and Norvig. Artificial Intelligence: A Modern Approach](http://aima.cs.berkeley.edu/). A comprehensive reference for all the AI topics that we will cover.
- [Koller and Friedman. Probabilistic Graphical Models](http://mitpress.mit.edu/books/probabilistic-graphical-models). Covers factor graphs and Bayesian networks (this is the textbook for CS228).
- [Sutton and Barto. Reinforcement Learning: An Introduction](https://mitpress.mit.edu/books/reinforcement-learning). Covers Markov decision processes and reinforcement learning (free online).
- [Hastie, Tibshirani, and Friedman. The Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn/). Covers machine learning from a rigorous statistical perspective (free online).
- [Tsang. Foundations of Constraint Satisfaction](http://www.bracil.net/edward/fcs.html). Covers constraint satisfaction problems (free online).

Note that some of these books use different notation and terminology from this course, so it may take some effort to make the appropriate connections.

CS 221 Artificial Intelligence: Principles and Techniques

Representations

Inference

Bayesian Networks Representation and Inference

Parameter Learning

Structure Learning

Simple Decisions

Exact Solutions

Approximate Value Functions

Online Planning

Policy Search

Policy Gradient Estimation and Optimization

Actor Critic

Exploration / Exploitation

Model-Based and Model-Free Reinforcement Learning

Beliefs

Exact Belief State Planning

Offline Belief State Planning

Online Belief State Planning

Controller Abstractions

Multiagent Reasoning

Policy Validation, Implications, and Summary

AA 228 / CS 238 Decision Making under Uncertainty Stanford University Winter 2023 The course introduces decision making under uncertainty from a computational perspective, covering dynamic programming, reinforcement learning, and more. Prerequisites include basic probability and fluency in a high-level programming language. This course introduces decision making under uncertainty from a computational perspective and provides an overview of the necessary tools for building autonomous and decision-support systems. Following an introduction to probabilistic models and decision theory, the course will cover computational methods for solving decision problems with stochastic dynamics, model uncertainty, and imperfect state information. Topics include Bayesian networks, influence diagrams, dynamic programming, reinforcement learning, and partially observable Markov decision processes. Applications cover air traffic control, aviation surveillance systems, autonomous vehicles, and robotic planetary exploration. - You will gain a broad fundamental understanding of the mathematical models and solution methods for decision making (exercises, three quizzes).
- You will be able to implement and extend key algorithms for learning and decision making (two programming projects).
- You will be able to identify an application of the theory in this course and formulate it mathematically (proposal).
- You will gain a deep understanding of an area of particular interest and apply it to a problem (final project).
- You will be able to critique approaches to solving decision problems (peer review). basic probability and fluency in a high-level programming language. This year we will be using the textbook titled *Algorithms for Decision Making*, MIT Press, 2022. Printed copies are available for purchase or you may download it for free as a PDF. Details are [here](https://algorithmsbook.com/decisionmaking).

The textbook serves as the official lecture notes.

AA 228 / CS 238 Decision Making under Uncertainty

TensorFlow

Model-based and model-free RL for robot control

Learning-based perception

Grasping and Manipulation

Learning-based grasping and manipulation

Interactive Perception

Imitation learning

Reinforcement Learning from Human Feedback (RLHF)

Interaction-aware learning

Shared autonomy

AA 174B / AA 274B / CS 237B / EE 260B Principles of Robot Autonomy II Stanford University Winter 2023 This course provides a deeper understanding of robot autonomy principles, focusing on learning new skills and physical interaction with the environment and humans. It requires familiarity with programming, ROS, and basic robot autonomy techniques. This course teaches advanced principles for endowing mobile autonomous robots with capabilities to autonomously learn new skills and to physically interact with the environment and with humans. Concepts that will be covered in the course are: Reinforcement Learning (RL) and its relationship to optimal control, contact and dynamics models for prehensile and non-prehensile robot manipulation, as well as imitation learning and human intent inference. Students will learn the theoretical foundations for these concepts. With this course, students will obtain a fundamental understanding of
advanced principles of robot autonomy, including robot learning, physical interaction with
the environment, and interaction with humans. - Familiarity with ROS and basic techniques for robot autonomy (e.g., AA274A or
equivalent).
- Familiarity with programming (e.g., CS 106A or equivalent) and Python.
- College calculus, linear algebra (e.g., CME 100 or equivalent).
- Basic probability and statistics (e.g., CME 106 or equivalent).  There is no required textbook.

AA 174B / AA 274B / CS 237B / EE 260B Principles of Robot Autonomy II

Tabular MDP planning

PyTorch

Tabular RL Policy Evaluation

Batch Reinforcement Learning

Value Alignment

CS 234: Reinforcement Learning Stanford University Winter 2023 This course offers a solid introduction to the field of reinforcement learning (RL), covering challenges, approaches, and deep RL. Prerequisites include Python proficiency and foundations of machine learning. Students will be able to implement RL algorithms and evaluate them. To realize the dreams and impact of AI requires autonomous systems that learn to make good decisions. Reinforcement learning is one powerful paradigm for doing so, and it is relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare. This class will provide a solid introduction to the field of reinforcement learning and students will learn about the core challenges and approaches, including generalization and exploration. Through a combination of lectures, and written and coding assignments, students will become well versed in key ideas and techniques for RL. Assignments will include the basics of reinforcement learning as well as deep reinforcement learning — an extremely promising new area that combines deep learning techniques with reinforcement learning. By the end of the class students should be able to:

- Define the key features of reinforcement learning that distinguishes it from AI and non-interactive machine learning (as assessed by the exam).
- Given an application problem (e.g. from computer vision, robotics, etc), decide if it should be formulated as a RL problem; if yes be able to define it formally (in terms of the state space, action space, dynamics and reward model), state what algorithm (from class) is best suited for addressing it and justify your answer (as assessed by the exam).
- Implement in code common RL algorithms (as assessed by the assignments).
- Describe (list and define) multiple criteria for analyzing RL algorithms and evaluate algorithms on these metrics: e.g. regret, sample complexity, computational complexity, empirical performance, convergence, etc (as assessed by assignments and the exam).
- Describe the exploration vs exploitation challenge and compare and contrast at least two approaches for addressing this challenge (in terms of performance, scalability, complexity of implementation, and theoretical guarantees) (as assessed by an assignment and the exam). - Proficiency in Python  
     All class assignments will be in Python. There is a tutorial [here](http://cs231n.github.io/python-numpy-tutorial/) for those who aren't as familiar with Python. If you have a lot of programming experience but in a different language (e.g. C/ C++/ Matlab/ Javascript) you will probably be fine.
- College Calculus, Linear Algebra (e.g. MATH 51, CME 100)  
     You should be comfortable taking derivatives and understanding matrix vector operations and notation.
- Basic Probability and Statistics (e.g. CS 109 or other stats course)  
     You should know basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
- Foundations of Machine Learning  
     We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. Either CS 221 or CS 229 cover this background. Some optimization tricks will be more intuitive with some knowledge of convex optimization. There is no official textbook for the class but a number of the supporting readings will come from:

- Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition. This is available for free [here](http://incompleteideas.net/book/the-book-2nd.html) and references will refer to the final pdf version available [here](http://incompleteideas.net/book/RLbook2018.pdf).

Some other additional references that may be useful are listed below:

- Reinforcement Learning: State-of-the-Art, Marco Wiering and Martijn van Otterlo, Eds. \[[link](https://link.springer.com/book/10.1007%2F978-3-642-27645-3)\]
- Artificial Intelligence: A Modern Approach, Stuart J. Russell and Peter Norvig.\[[link](http://aima.cs.berkeley.edu/)\]
- Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville. \[[link](http://www.deeplearningbook.org/)\]
- David Silver's course on Reinforcement Learning \[[link](http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching.html)\]

CS 234: Reinforcement Learning

The goal of Artificial Intelligence (AI) is to tackle complex real-world problems with rigorous mathematical tools. Common sub-topics include Machine Learning, Search, Markov Decision Processes, Reinforcement Learning, etc.

Artificial Intelligence

Computer Science

Artificial Intelligence

Prerequisites

Common concepts