¹ We will encourage three basic classes of final projects each one of which connects to both the DeepMind interactive agent model described in Abramson et al [2] and the programmer's apprentice application [9]. The first two classes of projects focus on, respectively, grounded language and neural programming and will emphasize implementation, i.e., designing, building, and evaluating neural network models. The third class of project focuses on the programmer's apprentice as a whole and will emphasize designing curriculum-training protocols and hybrid automated and scalable crowd-sourced data collection strategies to facilitate human-machine interaction and exploit the different roles of natural languages and programming languages in applications related to the programmer's apprentice.

Neural Programming

The neural programming community has grown substantially in the last few years. If you are interested in working on a coding project in this space, I recommend you begin by watching the video of Rishabh's talk at CVPR 2020 entitled Towards Human-like Program Synthesis and the paper [1] by Dan Abolafia, Rishabh, and their colleagues at Google Brain. As background, you might also read the neural programmer-interpreter (NPI) paper by Scott Reed and Nando de Freitas [24] (PDF) and the follow-up paper by Pierrot et al [23] extending NPI with recursive Monte Carlo Tree Search (PDF).

Rishabh and Dan along with Yash Savani will be participating in the class this quarter and are willing and interested in advising students working on neural programming coding projects. Yash recently graduated from Stanford, is currently working for Abacus AI and has been accepted to the graduate computer science programs at both CMU and Cornell. He has also been working with me and a small team of former CS379C students on neural programming and the programmer's apprentice project.

Grounding & Analogy

There is a growing interest in grounding language for its usefulness in reducing sample complexity, improving generalization, and providing support for analogical reasoning. Eve Clark has made significant contributions to our understanding of how children acquire their first language. Her research has underscored the critical importance of the interactive and collaborative aspects of language understanding including how infants and young children interact with their parents to establish a common ground for communication. Insights from her work are likely to be applicable to the curriculum learning protocols that the Interactive Agents Group at DeepMind are developing. I recommend that you look at Eve's recent book entitled First Language Acquisition, and her recent lectures and papers listed, respectively, under Resources, Lectures and Readings.

Analogical reasoning has not received a great deal of attention from the deep learning community until recently. I recommend that you begin by watching this video of Felix's lecture entitled Grounded Language Learning and Analogy: Developmental Pathway to Human Intelligence in Machines and reading the paper by Hill et al [14] (PDF) entitled Learning to Make Analogies by Contrasting Abstract Relational Structure that presents a promising approach to using deep learning for analogical reasoning.

Eve and Felix will be participating in the class this quarter and are interested in advising students working on projects related to grounding language, analogical reasoning, and the application of insights from linguistics and developmental cognitive science to training interactive agents such as the programmer's apprentice. As a source of inspiration for possible projects on grounding, you might take a look at some of the following papers:

Fu et al [10] from Sergey Levine's lab at UC Berkeley, Arumugam et al [5] from Stefanie Tellex's lab at Brown University, and Guadarrama et al [12] from Trevor Darrell, Pieter Abbeel and their colleagues and students also at Berkeley. Lampinen et al [18] (PDF) provide insight from a cognitive science perspective on the extent to which people actually use analogy and under what circumstances. The quickest way to get some idea if you want to read one of these papers is to read their BibTeX abstracts here, and then, if you're interested in reading the paper, you should be able to find it using the title as a query. If you have any problems, ask me and I will send it to you.

Interactive Intelligence

The third class of projects spans the entire programmer's apprentice application by focusing on methods for training interactive intelligent agents and building on the work of Abramson et al [2] to address the problems involved in agents that engage in symbolic behavior [25] of the sort required from the programmer's apprentice. Unlike the previous two classes, the projects in this class will not require writing code. Instead, the deliverables include one or more of the following: agent network architectures, curriculum-learning protocols, and scalable crowd-sourced training data acquisition strategies. Teams interested in pursuing projects in this class are encouraged to collaborate with teams working on the first two classes of projects in order to better understand the issues surrounding language-grounding and automated programming,

The emphasis of the projects in this class should be on applying the general concepts developed in the IAG paper and, the principles that follow from Eve Clark's observations and analysis of the way children learn their first language, to the design of a programmer's apprentice. Examples of interesting issues to address include whether to include, as part of the curriculum learning protocol, a variant of self-supervised motor babbling in which the agent generates random sequences of words before graduating to sequences in a simple pidgin language that observe enough structure to make them comprehensible – interpretable with a little effort on the part of the programmer/parent, but are not strictly grammatical.

The programmer and apprentice might also employ a simple sign language that would enable a form of shared attention, perhaps using an eye tracker to naturally draw attention to a portion of the monitor screen or using the mouse to specify an area of shared interest. Eve's description of self-initiated repair [8] (PDF) could provide the basis for designing a powerful new method of semi-supervised learning to shape the programmer's behavior in terms of providing useful feedback on the apprentice's efforts to communicate in both natural and programming language.

The marked-up version (PDF) of Abramson et al highlights the technologies employed in the curriculum used to train the IAG agent. The paper by Santoro et al [25] (PDF) describes the challenges and opportunities in building agents that engage in symbolic behaviors. If you're interested in this project, you will probably also want to familiarize yourself with recent developments in training reinforcement learning systems including hindsight experience replay [4], reinforcement learning with unsupervised auxiliary tasks [17], and curiosity-driven exploration by self-supervised prediction [22] – this fifteen-minute video provides a quick introduction to all three.

If you want to work on a project in the third class of projects, in addition to Eve and Felix, I expect Greg Wayne would be interested in hearing about and providing feedback on your ideas. Greg's research contributions on unsupervised predictive memory [27], robust imitation learning [26], hierarchical motor control [20], and dynamic external memory [11] are well worth your time exploring. You can find a link to a Stanford Libraries digital version of Eve's First Language Acquisition on the Resources page and I suggest that you first take a look at the summary that I put together. Recall that for copyright reasons these notes are access controlled; if you didn't hear or forgot the username and password that I gave out in the first class, just ask a fellow student or send me an email and I'll be glad to send it to you.

² Originally there were four categories, with the fourth being "biographical". The intent was to provide students with some background on my personal, educational and professional trajectory since in previous classes students have expressed curiosity and it is generally awkward to answer such questions in class. After some thought, I decided to relegate it to a footnote rather than have it intrude on the primary focus of this document. If you are interested in learning about my personal and professional background you can read about it here, and I would be happy to answer any lingering questions you might have outside of class time.