Distributed File Systems

The Internet

Classical synchronization

Concurrency

Time Synchronization

Distributed Mutual Exclusion

Remote Procedure Call (RPC)

Concurrency Control

Logging and Crash Recovery

Fault Tolerance

RAID (Redundant Array of Inexpensive Disks)

Distributed Replication

Distributed Databases

Spanner

Google File System (GFS)

Apache Hadoop Distributed File System (HDFS)

Data-Intensive Computing

MapReduce

Hadoop

In-memory cluster compute

Apache Spark

Distributed ML

Internet Content Delivery

DNS (Domain Name System)

Content Delivery Network (CDN)

Publish-Subscribe

Kafka

Virtualization

3-tier Web Service Architecture

Security Protocols

Byzantine Fault Tolerance

P2P Network

Distributed Ledgers

Blockchain

 15-440 Distributed Systems Carnegie Mellon University Fall 2020 A course offering both theoretical understanding and practical experience in distributed systems. Key themes include concurrency, scheduling, network communication, and security. Real-world protocols and paradigms like distributed filesystems, RPC, MapReduce are studied. Course utilizes C and Go programming languages. Most software is now distributed in some sense. This course is meant to serve as an introduction to distributed systems, emphasizing techniques for creating functional, usable, and high-performance distributed systems.

This course aims to: (1) provide students with an understanding of the principles and techniques behind the design of distributed systems, such as locking, concurrency, scheduling, and communication across the network. (2) provide students with practical experience designing, implementing, and debugging real distributed systems.

Major themes covered in this course include scarcity, scheduling, concurrency and concurrent programming, naming, abstraction and modularity, imperfect communication and other types of failure, protection from accidental and malicious harm, optimism, and the use of instrumentation and monitoring and debugging tools in problem solving. As the creation and management of software systems is a fundamental goal of any undergraduate systems course, students will design, implement, and debug large programming projects. After this course, students will have learned to…

- Implement and structure distributed systems programs.
- Write programs that can interoperate using well-defined protocols.
- Debug highly concurrent code that spans multiple programs running on multiple cores and machines.
- Reason about distributed algorithms for locking, synchronization and concurrency, scheduling, and replication.
- Use standard network communication primitives such as UDP and TCP.
- Understand the general properties of networked communication necessary for distributed systems programming in clusters and on the Internet.
- Employ and create common paradigms for easing the task of distributed systems programming, such as distributed filesystems, RPC, and MapReduce. Be able to clearly elucidate their benefits, drawbacks, and limitations.
- Identify the security challenges faced by distributed systems programs.
- Be able to select appropriate security solutions to meet the needs of commonly encountered distributed programming scenarios. Because this course has a big project component, you must be proficient in C and programming on UNIX systems. We will use the [Go programming language](https://golang.org/) throughout the term. It is required that you have taken 15-213 and gotten a “C-“ or higher since many of the programming skills you will need are taught in that course. **However, if you received a C in 15-213, you must meet with your academic advisor to discuss your background before taking 15-440/640, perhaps taking an additional course to sharpen your systems skills. Your advisor must email us approval and an explanation of why you have sufficient background to take 15-440/640.** 

 15-440 Distributed Systems

Parallelism and Distributed Systems

Thread (computing)

Interrupts

Input/output

Booting

Unix

Operating System Structure

Virtual Machine (VM)

Operating System Design

Microkernels

Scheduling (computing)

File system

Virtual memory

Operating System Security

Communication protocol

Avoiding Operating Systems

CS167 Operating Systems Brown University Spring 2023 CS167 offers comprehensive insights into the principles and intricacies of operating systems. Topics range from multithreaded programming to file system designs. Students will not only grasp theoretical knowledge but also get hands-on experience, particularly through the optional lab CS169, where they can develop an operating system called Weenix. In this class, you'll have the opportunity to learn both the overarching principles of operating systems as well as the intricacies of how they work.

We'll cover topics including multithreaded programming; managing threads, interrupts, and storage; processor scheduling; operating system structure; virtualization; security; and the design of both local and distributed file systems. You'll develop your skill as a C programmer while simultaneously learning what goes on inside your computer from the moment you power it on.

As part of the class, if you are enrolled in CS167, you may choose to take the (optional) half-credit lab, CS169. If you do so, you will attend the same lectures and complete the written assignments as usual. However, you will only complete the first two programming assignments from CS167, before embarking on an in-depth project wherein you build much of the functionality of a small operating system called Weenix. It is a significant time investment, and you'll still get a lot from the class if you don't take the lab. Some thoughts on taking CS169 from previous students can be found [here](). CS267 covers the material of both CS167 and 169; it is for graduate students only. ### The Course

- Implement threads
- Learn about operating systems
- Implement one (Weenix)
    - file-system-related components (1670)
    - most of the rest of it (1690, 2670) ### Skills Needed

- Ability to write and debug largish programs in C with POSIX threads
    - CSCI 330/1330 or CSCI 300/1310 (you may want to review POSIX threads material from 330 lectures)
- Basic computer architecture
    - CSCI 330/1330 or CSCI 300/1310 ### Workload Components

- Four moderate programming assignments
    - UThreads: simple user-level threads package
    - MThreads: more sophisticated user-level threads package
    - VFS: system calls and high-level file system
    - S5FS: low-level file system
- One large programming assignment: Weenix
- combines VFS and S5FS, as well as Processes, Drivers, and Virtual Memory
- Four homeworks
- In-class TopHat questions

### Textbook

Operating Systems In Depth: Design and Programming, 1st Edition

CS167 Operating Systems

Operating Systems

Comparison of distributed file systems

Distributed File Systems

2 courses cover this concept

15-440 Distributed Systems

CS167 Operating Systems