Grasping and Manipulation

Imitation learning

Interaction-aware learning

Interactive Perception

Learning-based grasping and manipulation

Learning-based perception

Markov Decision Process (MDP)

Model-based and model-free RL for robot control

Neural network

Reinforcement Learning from Human Feedback (RLHF)

Reinforcement learning (RL)

Shared autonomy

TensorFlow

AA 174B / AA 274B / CS 237B / EE 260B Principles of Robot Autonomy II Stanford University Winter 2023 This course provides a deeper understanding of robot autonomy principles, focusing on learning new skills and physical interaction with the environment and humans. It requires familiarity with programming, ROS, and basic robot autonomy techniques. This course teaches advanced principles for endowing mobile autonomous robots with capabilities to autonomously learn new skills and to physically interact with the environment and with humans. Concepts that will be covered in the course are: Reinforcement Learning (RL) and its relationship to optimal control, contact and dynamics models for prehensile and non-prehensile robot manipulation, as well as imitation learning and human intent inference. Students will learn the theoretical foundations for these concepts. With this course, students will obtain a fundamental understanding of
advanced principles of robot autonomy, including robot learning, physical interaction with
the environment, and interaction with humans. - Familiarity with ROS and basic techniques for robot autonomy (e.g., AA274A or
equivalent).
- Familiarity with programming (e.g., CS 106A or equivalent) and Python.
- College calculus, linear algebra (e.g., CME 100 or equivalent).
- Basic probability and statistics (e.g., CME 106 or equivalent).  There is no required textbook.

AA 174B / AA 274B / CS 237B / EE 260B Principles of Robot Autonomy II

The goal of Artificial Intelligence (AI) is to tackle complex real-world problems with rigorous mathematical tools. Common sub-topics include Machine Learning, Search, Markov Decision Processes, Reinforcement Learning, etc.

Artificial Intelligence

Computer Science

Robotics is a multidisciplinary field that combines elements of computer science, mechanical and electrical engineering, and more. In the realm of computer science, robotics is often focused on aspects such as robot programming, artificial intelligence (AI), machine learning, and autonomous systems. This can encompass things like perception (how a robot understands its environment), planning (how a robot decides what actions to take), and control (how a robot executes these actions).

Robotics

CS 234: Reinforcement Learning Stanford University Winter 2023 This course offers a solid introduction to the field of reinforcement learning (RL), covering challenges, approaches, and deep RL. Prerequisites include Python proficiency and foundations of machine learning. Students will be able to implement RL algorithms and evaluate them. To realize the dreams and impact of AI requires autonomous systems that learn to make good decisions. Reinforcement learning is one powerful paradigm for doing so, and it is relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare. This class will provide a solid introduction to the field of reinforcement learning and students will learn about the core challenges and approaches, including generalization and exploration. Through a combination of lectures, and written and coding assignments, students will become well versed in key ideas and techniques for RL. Assignments will include the basics of reinforcement learning as well as deep reinforcement learning — an extremely promising new area that combines deep learning techniques with reinforcement learning. By the end of the class students should be able to:

- Define the key features of reinforcement learning that distinguishes it from AI and non-interactive machine learning (as assessed by the exam).
- Given an application problem (e.g. from computer vision, robotics, etc), decide if it should be formulated as a RL problem; if yes be able to define it formally (in terms of the state space, action space, dynamics and reward model), state what algorithm (from class) is best suited for addressing it and justify your answer (as assessed by the exam).
- Implement in code common RL algorithms (as assessed by the assignments).
- Describe (list and define) multiple criteria for analyzing RL algorithms and evaluate algorithms on these metrics: e.g. regret, sample complexity, computational complexity, empirical performance, convergence, etc (as assessed by assignments and the exam).
- Describe the exploration vs exploitation challenge and compare and contrast at least two approaches for addressing this challenge (in terms of performance, scalability, complexity of implementation, and theoretical guarantees) (as assessed by an assignment and the exam). - Proficiency in Python  
     All class assignments will be in Python. There is a tutorial [here](http://cs231n.github.io/python-numpy-tutorial/) for those who aren't as familiar with Python. If you have a lot of programming experience but in a different language (e.g. C/ C++/ Matlab/ Javascript) you will probably be fine.
- College Calculus, Linear Algebra (e.g. MATH 51, CME 100)  
     You should be comfortable taking derivatives and understanding matrix vector operations and notation.
- Basic Probability and Statistics (e.g. CS 109 or other stats course)  
     You should know basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
- Foundations of Machine Learning  
     We will be formulating cost functions, taking derivatives and performing optimization with gradient descent. Either CS 221 or CS 229 cover this background. Some optimization tricks will be more intuitive with some knowledge of convex optimization. There is no official textbook for the class but a number of the supporting readings will come from:

- Reinforcement Learning: An Introduction, Sutton and Barto, 2nd Edition. This is available for free [here](http://incompleteideas.net/book/the-book-2nd.html) and references will refer to the final pdf version available [here](http://incompleteideas.net/book/RLbook2018.pdf).

Some other additional references that may be useful are listed below:

- Reinforcement Learning: State-of-the-Art, Marco Wiering and Martijn van Otterlo, Eds. \[[link](https://link.springer.com/book/10.1007%2F978-3-642-27645-3)\]
- Artificial Intelligence: A Modern Approach, Stuart J. Russell and Peter Norvig.\[[link](http://aima.cs.berkeley.edu/)\]
- Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville. \[[link](http://www.deeplearningbook.org/)\]
- David Silver's course on Reinforcement Learning \[[link](http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching.html)\]

CS 234: Reinforcement Learning

CS 271 / BIOMEDIN 220 Artificial Intelligence in Healthcare Stanford University Fall 2022-2023 Offered by Stanford University, this course focuses on AI applications in healthcare, exploring deep learning models for image, text, multimodal, and time-series data in the healthcare context. Topics also address AI integration challenges like interpretability and privacy. Healthcare is one of the most exciting application domains of artificial intelligence, with transformative potential in areas ranging from medical image analysis to electronic health records-based prediction and precision medicine. This course will involve a deep dive into recent advances in AI in healthcare, focusing in particular on deep learning approaches for healthcare problems. We will start from foundations of neural networks, and then study cutting-edge deep learning models in the context of a variety of healthcare data including image, text, multimodal and time-series data. In the latter part of the course, we will cover advanced topics on open challenges of integrating AI in a societal application such as healthcare, including interpretability, robustness, privacy and fairness. The course aims to provide students from diverse backgrounds with both conceptual understanding and practical grounding of cutting-edge research on AI in healthcare.  Proficiency in Python, or significant experience with a different programming language and ability to self-learn. Python will be used for homework assignments and the course project. Basic familiarity with college calculus (e.g. Math 19 or 41, comfortable taking derivatives), linear algebra (e.g. Math 51 or EE 103 / CME 103, comfortable with common matrix vector operations and notation), and probability and statistics (e.g. CME 106 or CS 109, comfortable with common probability distributions, mean, standard deviation, etc). Familiarity with machine learning, e.g. comfortable with the framework of machine learning and experience training a machine learning model. Familiarity with deep learning is highly recommended, e.g. prior experience training a deep learning model. 

CS 271 / BIOMEDIN 220 Artificial Intelligence in Healthcare

CS 221 Artificial Intelligence: Principles and Techniques Stanford University Autumn 2022-2023 Stanford's CS 221 course teaches foundational principles and practical implementation of AI systems. It covers machine learning, game playing, constraint satisfaction, graphical models, and logic. A rigorous course requiring solid foundational skills in programming, math, and probability. The goal of artificial intelligence (AI) is to tackle complex real-world problems with rigorous mathematical tools. In this course, you will learn the foundational principles and practice implementing various AI systems. Specific topics include machine learning, search, Markov decision processes, game playing, constraint satisfaction, graphical models, and logic.  This course is fast-paced and covers a lot of ground, so it is important that you have a solid foundation in a number of areas. Here are the basic skills that you need and the classes that teach those skills:

- Programming (ideally Python): [CS 106A](http://www.stanford.edu/class/cs106a/), [CS 106B](http://www.stanford.edu/class/cs106b/), [CS 107](http://www.stanford.edu/class/cs107/)
- Discrete math, mathematical rigor: [CS 103](http://www.stanford.edu/class/cs103/)
- Probability: [CS 109](http://www.stanford.edu/class/cs109/)
- Linear algebra: [Math 51](https://web.stanford.edu/class/math51/textbook.html)

It is less important that you know particular things (e.g., we don't use eigenvectors in this course even though that's a pillar of any linear algebra course), and more important that you've done enough related things that you feel at ease with it. While it is possible to fill in the gaps, this course does move quickly, and ideally you want to be focusing your energy on learning AI rather than catching up on prerequisites. We have made a few [prerequisite modules](https://stanford-cs221.github.io/autumn2022/modules) that you can review to refresh your memory, and the first homework (foundations) will allow you to also get some practice on these basics. ### Further Reading

There are no required textbooks for this class, and you should be able to learn everything from the lecture notes and homeworks. However, if you would like to pursue more advanced topics or get another perspective on the same material, here are some great resources:

- [Russell and Norvig. Artificial Intelligence: A Modern Approach](http://aima.cs.berkeley.edu/). A comprehensive reference for all the AI topics that we will cover.
- [Koller and Friedman. Probabilistic Graphical Models](http://mitpress.mit.edu/books/probabilistic-graphical-models). Covers factor graphs and Bayesian networks (this is the textbook for CS228).
- [Sutton and Barto. Reinforcement Learning: An Introduction](https://mitpress.mit.edu/books/reinforcement-learning). Covers Markov decision processes and reinforcement learning (free online).
- [Hastie, Tibshirani, and Friedman. The Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn/). Covers machine learning from a rigorous statistical perspective (free online).
- [Tsang. Foundations of Constraint Satisfaction](http://www.bracil.net/edward/fcs.html). Covers constraint satisfaction problems (free online).

Note that some of these books use different notation and terminology from this course, so it may take some effort to make the appropriate connections.

CS 221 Artificial Intelligence: Principles and Techniques

CS 188 Introduction to Artificial Intelligence UC Berkeley Fall 2022 UC Berkeley's CS 188 course covers the basic ideas and techniques for designing intelligent computer systems, emphasizing statistical and decision-theoretic modeling. By the course's end, students will have built autonomous agents that can make efficient decisions in a variety of settings. This course will introduce the basic ideas and techniques underlying the design of intelligent computer systems. A specific emphasis will be on the statistical and decision-theoretic modeling paradigm.

By the end of this course, you will have built autonomous agents that efficiently make decisions in fully informed, partially observable and adversarial settings. Your agents will draw inferences in uncertain environments and optimize actions for arbitrary reward structures. Your machine learning algorithms will classify handwritten digits and photographs. The techniques you learn in this course apply to a wide variety of artificial intelligence problems and will serve as the foundation for further study in any application area you choose to pursue.

See the syllabus for slides, deadlines, and the lecture schedule. Readings refer to [fourth edition of AIMA](http://aima.cs.berkeley.edu/) unless otherwise specified.  - **CS 61A or 61B**: Prior computer programming experience is expected (see below)
- **CS 70 or Math 55**: Familiarity with basic concepts of propositional logic and probability are expected (see below)

*CS61A AND CS61B AND CS70 is the recommended background.*

The required math background in the second half of the course will be significantly greater than the first half. The self-diagnostic assignment Homework 0 will help check your preparation.

Course programming assignments will be in Python. We do not assume that students have previous experience with the language, but we do expect you to learn the basics very rapidly. Project 0 is designed to teach you the basics of Python and how the CS 188 submission autograder works. Project 1 is a good representation of the programming level that will be required for subsequent projects in this class. 

AA 174B / AA 274B / CS 237B / EE 260B Principles of Robot Autonomy II

Overview

Prerequisites

Learning objectives

Textbooks and other notes

Other courses in Artificial Intelligence

CS 234: Reinforcement Learning

CS 271 / BIOMEDIN 220 Artificial Intelligence in Healthcare

CS 221 Artificial Intelligence: Principles and Techniques

CS 188 Introduction to Artificial Intelligence

Courseware availability

Covered concepts