List of Topics in Order of Presentation

Functional Components:

  1. Actor Critic Learning (basal ganglia and reinforcement learning) — April 14, Tuesday

    Speaker: Randy O'Reilly
    Affiliation: University of California Davis
    Readings: O'Reilly et al [27] Chapter 7 Motor Control and Reinforcement Learning
    Date: April 12, 2018
    Venue: Stanford, Invited Class Discussion in CS379C Computational Models of the Neocortex
    Title: "Programming in the Brain" [Discussion on Reinforcement Learning] (SLIDES) (VIDEO)

    Supplements1:

    Speaker: Matt Botvinick
    Affiliation: Deepmind & University College London
    Readings: Wang et al [37]
    Date: April 26, 2018
    Venue: Stanford, Invited Class Discussion in CS379C Computational Models of the Neocortex
    Title: "Prefrontal cortex as a meta-reinforcement learning system" (SLIDES) (VIDEO)

  2. Episodic Memory (hippocampus and differentiable neural computers) — April 16, Thursday

    Speaker: Randy O'Reilly
    Affiliation: University of California Davis
    Readings: O'Reilly et al [27] Chapter 8 Learning and Memory
    Date: April 12, 2018
    Venue: Stanford, Invited Class Discussion in CS379C Computational Models of the Neocortex
    Title: "Programming in the Brain" [Discussion on Hippocampal Formation] (SLIDES) (VIDEO)

    Supplements:

    Speaker: Alex Graves
    Affiliation: Google Deepmind
    Readings: Graves et al [18, 17], Weston et al [35, 38]
    Date: March 22, 2017
    Venue: NIPS 2016 Symposium on Recurrent Neural Networks
    Title: "Differentiable Neural Computers" (VIDEO)

    Speaker: Sam Gershman
    Affiliation: Harvard University
    Readings: Gershman et al [16, 34]
    Date: April 22, 2019
    Venue: Harvard Center of Mathematical Sciences and Applications
    Title: "The Hippocampus As a Predictive Map" (VIDEO)

    Related Topics:
    Hippocampal Formation [32, 24, 25]

  3. Action Selection (frontal cortex, striatum, thalamus and working memory) — April 21, Tuesday

    Speaker: Randy O'Reilly
    Affiliation: University of California Davis
    Readings: O'Reilly et al [27] Chapter 10 Executive Function
    Date: February 22, 2020
    Venue: Stanford, Strong AI Group Discussion in Wu Tsai Neurosciences Institute
    Title: "Prefrontal Cortex Basal Ganglia Working Memory" (TRANSCRIPT)

    Supplements:

    Speaker: Matt Botvinick
    Affiliation: Deepmind & University College London
    Readings: Botvinick et al [6], Santoro et al [31]
    Date: December 16, 2018
    Venue: Stanford University, Natural / Artificial Intelligence Symposium
    Title: "Reinforcement learning: Fast and Slow" (VIDEO)

    Related Topics:
    Complementary Learning Systems [22, 26, 20]

  4. Executive Control (prefrontal cortex, planning complex cognitive behavior) — April 23, Thursday

    Author: Joaquín M. Fuster
    Affiliation: University of California Los Angeles
    Readings: Fuster [13], Merel et al [23], Botvinick et al [7, 8], Frank and Badre [3]
    Title: "Hierarchical Structure in the Prefrontal Cortex" (PDF)

    Speaker: Yoshua Bengio3
    Affiliation: University of Montreal
    Readings: Bengio et al [30, 5]
    Date: February 4, 2010
    Venue: Samsung AI Forum
    Title: "Towards Compositional Understanding of the World by Deep Learning" - VIDEO

    Speaker: Chelsea Finn
    Affiliation: Stanford University
    Readings: Finn et al [11, 33]
    Date: August 5, 1019
    Venue: Simons Institute - Emerging Challenges in Deep Learning
    Title: "Flexible Neural Networks and the Frontiers of Meta-Learning" (VIDEO)

    Speaker: Sergey Levine
    Affiliation: University of California Berkley
    Readings: Levine et al [36, 10, 2]
    Date: November 8, 2019
    Venue: Institute for Advanced Study - Workshop on New Directions in Reinforcement Learning and Control
    Title: "Deep Reinforcement Learning in the Real World" (VIDEO)

    Related Topics:
    Deep Learning Framework for Neuroscience [29]

  5. Prediction Action Cycle (reciprocally connected motor and sensory systems) — April 28, Tuesday

    Speaker: David Cox
    Affiliation: Harvard University, IBM
    Readings: Lotter, Kreiman and Cox [21], Alexander and Brown [1], van den Oord et al [39]
    Date: Apr 16, 2018
    Venue: Simons Institute - Computational Theories of the Brain
    Title: "Predictive Coding Models of Perception" (VIDEO)

    Speaker: Karl Friston
    Affiliation: University College London
    Readings: Friston [12], Rao and Ballard [28]
    Date: September 2, 2016
    Venue: Center for Cognitive Neuroscience Workshop: Predictive Coding
    Title: "Predictive Coding, Active Inference and Belief Propagation" (VIDEO)

    Speaker: Sam Gershman
    Affiliation: Harvard University
    Readings: Gershman [15]
    Date: February 19, 2019
    Venue: Brain Inspired - A podcast on the convergence of neuroscience and artificial intelligence
    Title: "Free Energy Principle & Human Machines" (AUDIO)

    Speaker: Jun-Yan Zhu
    Affiliation: University of California Berkley
    Readings: Zhu et al [40, 14]
    Date: November 24, 2017
    Venue: International Conference on Computer Vision
    Title: "Tutorial on Generative Adversarial Networks - CycleGAN" (VIDEO)

    Related Topics:
    Helmholtz Machines [Tutorial] — Kirby [19]
    Variational Autoencoders [Tutorial] — Doersch [9]

Basic Representations:

  1. Relational Learning (recognizing, learning and using relational models) — April 30, Thursday

  2. Graph Networks (learning, reasoning about connections between entities) — May 5, Tuesday

  3. Interaction Networks (learning and reasoning about dynamical systems) — May 7, Thursday

  4. Analogical Reasoning (combining existing models to create new models) — May 12, Tuesday

  5. Perception As Memory (attention, affordances, anticipatory behaviors) — May 14, Thursday

Fundamental Problems:

  1. Algorithmic Abstraction (when we generalize skills creating hierarchies) — May 19, Tuesday

  2. Catastrophic Forgetting (when we forget old skills in learning new ones) — May 21, Thursday

  3. Partial Observability (when the system is a higher-order Markov process) — May 26, Tuesday:

References

[1]   Joshua W. Alexander, William H.and Brown. Frontal cortex function as derived from hierarchical predictive coding. Scientific Reports, 8:3843, 2018.

[2]   Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. CoRR, arXiv:1611.01796, 2016.

[3]   D. Badre and M. J. Frank. Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits II: evidence from fMRI. Cerebral Cortex, 22(3):527--536, 2012.

[4]   Mark F. Bear, Barry Connors, and Michael Paradiso. Neuroscience: Exploring the Brain (Third Edition). Lippincott Williams & Wilkins, Baltimore, Maryland, 2006.

[5]   Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798--1828, 2013.

[6]   Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in Cognitive Sciences, 23:408--422, 2019.

[7]   Matthew M. Botvinick. Multilevel structure in behaviour and in the brain: a model of fuster's hierarchy. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362:1615--1626, 2007.

[8]   Matthew M. Botvinick, Yael Niv, and Andrew C. Barto. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 113(3):262--280, 2009.

[9]   Carl Doersch. Tutorial on variational autoencoders. CoRR, arXiv:1606.05908, 2017.

[10]   Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex X. Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. CoRR, arXiv:1812.00568, 2018.

[11]   Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, arXiv:1703.03400, 2017.

[12]   Karl Friston. Does predictive coding have a future? Nature Neuroscience, 21, 2018.

[13]   Joaquín M. Fuster. Chapter 8: An Overview of Prefrontal Functions, pages 375--425. Elsevier, London, 2015.

[14]   Bruno Gavranovic. Compositional deep learning. CoRR, arXiv:1907.08292, 2019.

[15]   Samuel J. Gershman. What does the free energy principle tell us about the brain? CoRR, arXiv:1901.07945, 2019.

[16]   Samuel J. Gershman and Nathaniel D. Daw. Reinforcement learning and episodic memory in humans and animals: An integrative framework. Annual Reviews of Psychology, 68:101--128, 2017.

[17]   Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. CoRR, arXiv:1410.5401, 2014.

[18]   Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdoménech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 538:471--476, 2016.

[19]   Kevin G. Kirby. A tutorial on helmholtz machines. Department of Computer Science, Northern Kentucky University, 2019.

[20]   Dharshan Kumaran and James L. McClelland. Generalization through the recurrent interaction of episodic memories: A model of the hippocampal system. Psychology Review, 119:573--616, 2012.

[21]   William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. Proceedings of the 5th International Conference on Learning Representations, 2017.

[22]   James L. McClelland, Bruce L. McNaughton, and Andrew K. Lampinen. Integration of new information in memory: New insights from a complementary learning systems perspective. Submitted to the Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 2019.

[23]   J. Merel, M. Botvinick, and G. Wayne. Hierarchical motor control in mammals and machines. Nature Communications, 10(1):5489, 2019.

[24]   K. A. Norman. How hippocampus and cortex contribute to recognition memory: Revisiting the complementary learning systems model. Hippocampus, 20(11):1217--1227, 2010.

[25]   Kenneth Norman and Randall O'Reilly. Modeling hippocampal and neocortical contributions to recognition memory: A complementary learning systems approach. Psychological review, 110:611--46, 2003.

[26]   Randall C. O'Reilly, Rajan Bhattacharyya, Michael D. Howard, and Nicholas Ketz. Complementary learning systems. Cognitive Science, 38(6):1229--1248, 2014.

[27]   Randall C. O'Reilly, Yuko Munakata, Michael J. Frank, Thomas E. Hazy, and Contributors. Computational Cognitive Neuroscience. Wiki Book, Third Edition, 2016.

[28]   Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2:79--87, 1999.

[29]   Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lindsay, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, JoÃo Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording. A deep learning framework for neuroscience. Nature Neuroscience, 22:1761--1770, 2019.

[30]   Jake Russin, Jason Jo, Randall C. O'Reilly, and Yoshua Bengio. Compositional generalization in a deep seq2seq model by separating syntax and semantics. CoRR, arXiv:1904.09708, 2019.

[31]   Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1842--1850, New York, New York, USA, 2016. PMLR.

[32]   Anna C. Schapiro, Nicholas B. Turk-Browne, Matthew M. Botvinick, and Kenneth A. Norman. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1711):20160049, 2017.

[33]   Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. CoRR, arXiv:1804.00645, 2018.

[34]   Kimberly L. Stachenfeld, Matthew M. Botvinick, and Samuel J. Gershman. The hippocampus as a predictive map. Nature Neuroscience, 20:1643, 2017.

[35]   Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2440--2448. Curran Associates, Inc., 2015.

[36]   Aviv Tamar, Sergey Levine, and Pieter Abbeel. Value iteration networks. CoRR, arXiv:1602.02867, 2016.

[37]   Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. bioRxiv, 2018.

[38]   Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. CoRR, arXiv:1410.3916, 2014.

[39]   Representation Learning with Contrastive Predictive Coding. Aaron van den oord and yazhe li and oriol vinyals. CoRR, arXiv:1807.03748, 2018.

[40]   Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR, arXiv:1703.10593, 2017.

1 I've selected short descriptions of some of the core concepts that we will be talking about in class. They are not nearly as detailed as those provided in the recommended text by O'Reilly  [27], which is not as comprehensive as an introductory neuroscience such as Bear, Connors and Paradiso [4]. That said, this class is not about teaching you basic neuroscience, but rather about teaching you how to work with and learn from neuroscientists in collaborative, multi-disciplinary science and engineering teams of the sort that you will encounter at Deepmind and Google Brain as well as a growing number of academic institutions. Think of these short videos as a means of bootstrapping or refreshing your knowledge of neuroscience to prepare for working in such teams. There are similar resources on the Khan Academy web site offering a wide range of courses covering introductory to advanced topics.

2 The video is largely guided by Donald Hebb's "The Organization of Behaviour" (1949), which has served as the foundation for our current understanding of brain-behavior relationships and synaptic plasticity. It outlines Hebb's three postulates to guide a multi-level understanding of the mechanisms by which the nervous system stores and interacts with information. Hebb’s three postulates, with particular attention to the third postulate – "the phase sequence" and "stream of consciousness," are explored as events occurring within the human brain in response to external stimuli. The Hebbian synapse and the cell assembly are explained in light of the long term potentiation (LTP) work done by Tim Bliss and Terje Lomo (1973), and Bruce McNaughton (1978). SOURCE: Updated final version of "Hebb's Three Postulates: from Brain to Soma": Barsen, S. (Director, Designer, Animator), Barnes, S. J. (Producer), Ritland, L. (Editor). (2015, Oct 18). Animating Hebb’s Three Postulates: from Brain to Soma [hebb.metaplasticity.com].

3 Humans are much better than current AI systems at generalizing out-of-distribution. What ingredients can bring us closer to that level of competence? We propose four ingredients combined: (a) meta-learning (to learn end-to-end to generalize to modified distributions, sampled from a distribution over distributions), (b) designing modular architectures with the property that modules are fairly independent of each other and interacting sparsely while made to be composed in new ways easily, (c) capturing causal structure decomposed into independent mechanisms so as to correctly infer the effect of interventions by agents which modify the data distribution, and (d) building better and more stable models of the invariant properties of possibly changing environments by taking advantage of the interactions between the learner and its environment to learn semantic high-level variables and their interactions, i.e., adopting an agent perspective on learning to benefit deep learning of abstract representations. The last ingredient implies that learning purely from text is not sufficient and we need to strive for learning agents which build a model of the world, to which linguistic labels can be associated, i.e., performing grounded language learning. Whereas this agent perspective is reminiscent of deep reinforcement learning, the focus is not on how deep learning can help reinforcement learning (as a function approximation black box) but rather how the agent perspective common in reinforcement learning can help deep learning discover better representations of knowledge.