Books and expository writingJohn Duchi, Stanford University
Information Theory, Learning, and StatisticsInformation Theory, Learning, and Statistics
blurb: While information theory solves fundamental problems in the theory of communications, its connections to statistical estimation, inference, and machine learning date to the birth of the field. With their focus on fundamental limits, information theoretic techniques provide deep insights into numerous optimal procedures. In addition, the basic quantities of information theory–entropy and divergence measures– underpin many areas of mathematical statistics and probability. Using these connections, this textbook explores the breadth of interactions between information theory and modern statistics, machine learning, and optimization: can we develop prediction schemes with guaranteed accuracy? How can we develop optimal estimators that provide privacy of individuals providing data? How do entropy and other measures connect with prediction and online game playing? This textbook develops the tools to answer such questions. Moreover, many of the most important results on optimality rely on hard-to-find convex analytic techniques; the book develops these both with illustrations and from first principles. Introductory Lectures on Stochastic Convex OptimizationIntroductory Lectures on Stochastic Convex Optimization |