CME 323: Distributed Algorithms and Optimization
Spring 2018, Stanford University
Tue, Thu 12:00 PM  1:20 PM at 260113 (04/02/2018  06/06/2018)
Instructor: Reza Zadeh
Computer Science is evolving to utilize new hardware such as GPUs, TPUs, CPUs, and large commodity clusters thereof.
Many subfields such as Machine Learning and Optimization have adapted their algorithms to handle such clusters.
Topics include distributed and parallel algorithms for: Optimization, Numerical Linear Algebra, Machine Learning, Graph analysis, Streaming algorithms, and other problems that are challenging to scale on a commodity cluster.
The class will focus on analyzing programs, with some implementation using Apache Spark and TensorFlow.
The course will be split into two parts: first, an introduction
to fundamentals of parallel algorithms and runtime analysis on a single multicore machine.
Second, we will cover distributed algorithms running on a cluster of machines.
Class Format
We will focus on the analysis of parallelism and distribution costs of algorithms.
Sometimes, topics will be illustrated with exercises
using Apache Spark and TensorFlow.
Prerequisites: Targeting graduate students having
taken Algorithms at the level of CME 305 or CS 161.
Being able to competently program in any mainstream high level language.
There will be homeworks, a midterm, and a final exam.
Grade Breakdown:
Homeworks: 40%
Midterm: 30%
Final: 30%
The midterm will be in class on Tuesday May 8th.
Textbooks:
Parallel Algorithms
by Guy E. Blelloch and Bruce M. Maggs [BB]
Introduction to Algorithms by Cormen, Leiserson, Rivest, Stein [CLRS]
Learning Spark
by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia [KKWZ]
TensorFlow for Deep Learning
by Bharath Ramsundar and Reza Zadeh [RZ]
Homework
Homework 1 [pdf] [tex] Due April 19th. [soln]
Homework 2 [pdf] [tex] Due May 3rd. [soln]
Homework 3 [pdf] [tex] Due May 22nd. [soln]
Homework 4 [pdf] [tex] Due June 5th. [soln]
Lectures and References
 Lecture 1: Fundamentals of Distributed and Parallel algorithm analysis. Reading: BB Chapter 1.
Lecture Notes
 Lecture 2: Scalable algorithms, Scheduling. Reading: BB 5.
Lecture Notes,
Handbook of Scheduling,
Graham's Algorithm,
TensorFlow Scheduling,
Better Bounds for Online Scheduling
 Lecture 3: Prefix Sum, Mergesort. Reading: KT 5, BB 8.
Lecture Notes,
Cole's parallel merge sort (1988)
 Lecture 4: Introduction to TensorFlow.
Guest lecture by Bharath Ramsundar. Lecture Slides
 Lecture 5: Parallel quickselect, Quicksort, Strassen's Algorithm, Minimum Spanning Trees. Reading: KT 3, 4.5, 4.6.
Lecture Notes (a), Lecture Notes (b),
Linear time bounds for median select,
Prefix scan qsort,
Boruvka (1926).
 Lecture 6: Graph contraction, star contraction, MST algorithms. Reading: CLRS 12, 13.
Lecture Notes.
 Lecture 7: (Stochastic) Gradient Descent, Parallel SGD (HOGWILD!). HOGWILD!, Omnivore.
Lecture Notes.
 Lecture 8: Intro to distributed computing, sampling, communication patterns.
Lecture Notes.
 Lecture 9: Distributed sort, intro to map reduce, applications to map reduce.
 Lecture 10: Map Reduce (indexing), Sparse Matrix Multiplies using SQL, Joins using Map Reduce.
 Lecture 11: Midterm
 Lecture 12: Joins using map reduce, measures of complexity, triangle counting.
Curse of the Last Reducer.
 Lecture 13: Sparse matrix multiplication, triangle counting, measures of complexity.
Sparse matrix multiplication using SQL, Sparse matrix multiplication in MapReduce.
 Lecture 14: Intro to Spark and other Distributed Computing Tools, Gradient Descent in Spark, Communication Patterns
Intro to Spark, Intro to Distributed Optimization, Communication Patterns.
 Lecture 15: Data Flow Systems: Spark, MapReduce shortcomings. Broadcasting. Partitioning for PageRank.
Intro to Spark, Communication Patterns, Partitioning for Pagerank.
 Lecture 16: Communication Patterns, Partitioning for PageRank, Singular Value Decomposition.
Communication Patterns, Partitioning for Pagerank, Singular Value Decomposition.
 Lecture 17: Covariance Matrices and Allpairs similarity
Covariance Matrices and Allpairs similarity.
 Lecture 18: Spark vs tensorflow, Final review.
Previous Years
Spring 2015: [class webpage]
Spring 2016: [class webpage]
Spring 2017: [class webpage]



Contact
Reza: rezab at stanford
Office hours: by appointment
TA
Yokila Arora: yarora at stanford
Office hours: Monday 122, Wednesday 24.
TA office hours will be held in the Huang Engineering Center basement
(in front of the ICME office)

