Automatic differentiation

Learning with Adversaries

Machine Learning Life-cycle

Prediction Serving

Distributed Model Training

Autonomous Driving

Database Systems and Machine Learning

Model Compilation

Machine Learning Frameworks

Hardware Acceleration for Machine Learning

AI-Sys Machine Learning Systems UC Berkeley Fall 2019 A course focused on the intersection of AI and systems, it discusses trends in system designs and AI applications for optimizing architecture and performance of systems. It requires background in system design or machine learning and involves hands-on projects. The recent success of AI has been in large part due in part to advances in hardware and software systems. These systems have enabled training increasingly complex models on ever larger datasets. In the process, these systems have also simplified model development, enabling the rapid growth in the machine learning community. These new hardware and software systems include a new generation of GPUs and hardware accelerators (e.g., TPU and Nervana), open source frameworks such as Theano, TensorFlow, PyTorch, MXNet, Apache Spark, Clipper, Horovod, and Ray, and a myriad of systems deployed internally at companies just to name a few. At the same time, we are witnessing a flurry of ML/RL applications to improve hardware and system designs, job scheduling, program synthesis, and circuit layouts.

In this course, we will describe the latest trends in systems designs to better support the next generation of AI applications, and applications of AI to optimize the architecture and the performance of systems. The format of this course will be a mix of lectures, seminar-style discussions, and student presentations. Students will be responsible for paper readings, and completing a hands-on project. For projects, we will strongly encourage teams that contains both AI and systems students. - Identify and solve impactful problems at the intersection of AI and Systems
- Learn about the big ideas and key results in AI systems
- Learn how to read, evaluate, and peer review papers
- Learn how to write papers that get high review scores - You have taken **previous courses** in either
    - Systems (for example CS162) → comfortable **building systems** and familiar with **big ideas in system design**
    - Machine Learning (for example CS189) → comfortable **training 
neural networks** and familiar with **big ideas in ML**
- You are **actively involved in research** in either systems or 
machine learning (or both) ### Reasons not to take this class … 

- If you want to learn how to **train models** and **use TensorFlow, PyTorch, and SkLearn**
    - Take: CS C100, CS182, CS189
- If you want learn how to **use big data systems**
    - Take: CS162, CS186
- You are not interested in learning how to **evaluate, conduct**, and **communicate** research
    - Why not?
- You are **unable to attend lecture**

AI-Sys Machine Learning Systems

Artificial Intelligence

Machine Learning

Supervised learning

Classification

Linear regression

Training/Validation/Testing

Perceptron

MNIST

Loss function

Optimization

Backpropagation

Matrix representation of Neural Networks

Graphics Processor (GPU)

TensorFlow

Multilayer perceptron

Activation Functions

Convolutional neural network (CNN)

Overfitting

Regularization (mathematics)

Language Models

Word Embeddings

Feedforward neural network

Recurrent neural network (RNN)

Long Short-Term Memory (LSTM)

Gated recurrent unit (GRU)

Machine Translation

Attention (machine learning)

Transformer (machine learning model)

Scaling Deep Learning Systems

Interpretation of Neural Networks

Unsupervised learning

Autoencoders

Variational autoencoder (VAE)

Generative adversarial network (GAN)

Deepfake

Reinforcement learning (RL)

Value iteration

Deep Q-learning

Policy Gradient Methods

Actor-Critic Methods

Graph neural network (GNN)

CSCI 1470/2470 Deep Learning Brown University Spring 2022 Brown University's Deep Learning course acquaints students with the transformative capabilities of deep neural networks in computer vision, NLP, and reinforcement learning. Using the TensorFlow framework, topics like CNNs, RNNs, deepfakes, and reinforcement learning are addressed, with an emphasis on ethical applications and potential societal impacts. Over the past few years, Deep Learning has become a popular area, with deep neural network methods obtaining state-of-the-art results on applications in computer vision (Self-Driving Cars), natural language processing (Google Translate), and reinforcement learning (AlphaGo). These technologies are having transformative effects on our society, including some undesirable ones (e.g. deep fakes).

This course is there to give students a practical understanding of how Deep Learning works, how to implement neural networks, and how to apply them ethically. We introduce students to the core concepts of deep neural networks and survey the techniques used to model complex processes within the contexts of computer vision and natural language processing.

Throughout the course, we emphasize and require students to think critically about potential ethical pitfalls that can result from mis-application of these powerful models. The course is taught using the Tensorflow deep learning framework.
 By the end of this course, you will be able to:

- Learn about the fundamental algorithms that underly all modern deep learning models.
- Implement different types of deep learning models in Tensorflow.
- Think critically about using a deep learning model for a task and its potential societal impact.
- Collaborate with classmates on a team project to apply deep learning models to task of your
choice.
- Communicate your findings (both positive and negative results are encouraged) through pre-
sentations. - A basic programming course: (CSCI 0150, 0170 or 0190)
- A linear algebra course: (CSCI 0530, MATH 0520 or 0540)
- A stats / probability course: (CSCI 0220, 1450, 0450, MATH 1610, APMA 1650 or 1655)

Exceptions may be possible for those missing one of these prerequisites if (a) the student has taken
another course which covers similar material, or if (b) the student will be concurrently taking the
prerequisite. If either of these situations applies to you, use the “Request Override” feature in
Courses@Brown to request an override code (and explain why you believe your situation merits
one). ### Textbook

None required. Students are encouraged to refer to the following textbook, which is available online:

- [Deep Learning](https://www.deeplearningbook.org/), by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

CSCI 1470/2470 Deep Learning

Deep Learning

Automatic differentiation

2 courses cover this concept

AI-Sys Machine Learning Systems

CSCI 1470/2470 Deep Learning