CME 323: Distributed Algorithms and Optimization

Computer Science is evolving to utilize new hardware such as GPUs, TPUs, CPUs, and large commodity clusters thereof. Many subfields such as Machine Learning and Optimization have adapted their algorithms to handle such clusters.

Topics include distributed and parallel algorithms for: Optimization, Numerical Linear Algebra, Machine Learning, Graph analysis, Streaming algorithms, and other problems that are challenging to scale on a commodity cluster. The class will focus on analyzing programs, with some implementation using Apache Spark and TensorFlow.

The course will be split into two parts: first, an introduction to fundamentals of parallel algorithms and runtime analysis on a single multicore machine. Second, we will cover distributed algorithms running on a cluster of machines.

Class Format

We will focus on the analysis of parallelism and distribution costs of algorithms. Sometimes, topics will be illustrated with exercises using Apache Spark and TensorFlow.

Pre-requisites: Targeting graduate students having taken Algorithms at the level of CME 305 or CS 161. Being able to competently program in any main-stream high level language. There will be homeworks, a midterm, and a final exam.

Grade Breakdown:
Homeworks: 40%
Midterm: 30%
Final: 30%

Textbooks:
Parallel Algorithms by Guy E. Blelloch and Bruce M. Maggs [BB]
Introduction to Algorithms by Cormen, Leiserson, Rivest, Stein [CLRS]
Learning Spark by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia [KKWZ]
TensorFlow for Deep Learning by Bharath Ramsundar and Reza Zadeh [RZ]
Algorithm Design by Kleinberg and Tardos [KT]

Logistics

Homeworks will be assigned via Ed and due on Gradescope.

We will be hosting office hours in Huang B019 and simultaneously open a Zoom session for those unable to attend in person. However, we encourage students to post questions publicly on Ed.

The midterm and final exams will be held exclusively in class. If you encounter any scheduling conflicts, please reach out to the teaching staff to arrange accommodations.

Homework

Homework 1 [pdf][tex] Due April 18th

Lectures and References

• Lecture 1 (4/2): Introduction to Parallel Algorithms (PRAM Model, Work + Depth, Computation DAGs, Brent's Theorem, Parallel Summation)
  Lecture 1
  
• Lecture 2 (4/4): Scalability, Scheduling, All Prefix Sum Reading: BB 5.
  Lecture 2, Handbook on Scheduling, Graham's Algorithm, TensorFlow Scheduling, Better Bounds for Online Scheduling
  
• Lecture 3 (4/9): All Prefix Sum, Mergesort. Reading: KT 5, BB 8.
  Lecture 3, Thinking in Parallel: Some Basic Data-Parallel Algorithms and Techniques, Cole's parallel merge sort (1988)
  
• Lecture 4 (4/11): Divide and Conquer Algorithms, Master Theorem, Quick Selection, Quick Sort.
  Lecture 4, Linear time bounds for median select, Prefix scan qsort
  
• Lecture 5 (4/16): Matrix Multiplication (Strassen's Algorithm).
  Lecture 5
• Lecture 6 (4/18): Graph Algorithms, Star Contraction, Minimum Spanning Tree (Boruvka's Algorithm. Reading: KT 3, 4.5, 4.6, CLRS 12, 13.
  Lecture 6, Boruvka (1926)
  
• Lecture 7 (4/23): Solving Linear Systems
  Lecture 7
  
• Lecture 8 (4/25): Optimization for Machine Learning, HOGWILD!
  Lecture 8, HOGWILD!, Omnivore.
  

Previous Years

Spring 2015: [class webpage]

Spring 2016: [class webpage]

Spring 2017: [class webpage]

Spring 2018: [class webpage]

Spring 2020: [class webpage]

Spring 2022: [class webpage]