My Twitter Experience

When?
I spent the academic year of 2009-10 (and the two sandwiching summers) at Twitter as a consulting research fellow. Subsequently, during the next four years, I spent a day a week at Twitter, serving as a technical advisor and research fellow.


What?
My first year at Twitter was quite productive. I prototyped a number of algorithmic products, including Twitter's recommendation system, their ad targeting, and their explore-exploit method for ads. I also helped the company realize that tweets could double as ads, which led to a very successful promoted tweets product. Later, I also helped with recruitment, with figuring out the computational architecture for several products, and with ongoing improvements in search, recommendation, and monetization algorithms. I helped craft Twitter's research strategy.

My WWW2014 talk explains some of my contributions at Twitter; if you don't have keynote, you can access the pdf version.


Why?
In 2009, I felt that there was very little I could do at another University or research lab that I couldn't at Stanford; I had talked to only a fraction of my colleagues here, so it wasn't clear how going somewhere else would be a big help to my research. At the same time, I had been bothered by the fact that several of my ideas which I had thought were worthy of consideration elicited a "ho-hum" response from industry -- my work with Kamesh Munagala on hybrid auctions, my work with Pankaj Gupta and others on TCAM-based algorithms, my work on Pagerank and recommendation markets, etc. It seemed clear that the best way forward in having an algorithmic impact was to go somewhere with a clear need for algorithmic products.


Was it worth it?

It was very satisfying to see algorithms converted into practice, touch millions of users, and help monetize a service that has become a societal force. Twitter was small, but the speed at which it executed in converting algorithmic prototypes into products surprised me. Part of this was the amazing executives, PMs, and engineers I was working with at the time (Dick Costolo, Anamitra Banerjee, and Utkarsh Srivastava on ads; Ev Williams, Josh Elman, Pankaj Gupta, Abdur Chowdhury, and John Sirois on recommendations). But a part was definitlely timing. I had the option to go to Facebook for a similar position. Facebook had by far the better infrastructure and the bigger engineering team in 2009, but they were well down the path of both recommendations and monetization.

The experience also shaped my research in three ways. First, it gave me research problems to work on. Three papers I am immensely proud of (Incremental PageRank with Abdur and Bahman in 2011; Precision of social networks with Aneesh, Kamesh, and Reza in 2013; and Fast Personalized PageRank with Peter, Seshadhri, and Sid, in 2014) came about because I had a better appreciation of the right research questions to ask, from both a theory and a practical angle. Second, it gave me confidence to work on big problems, leading to me jumping into crowdsourced democracy. Third, it helped me realize how important team work is -- I expect more of my students and collaborators now and hold them accountable, but also place more trust in their ability and judgement.



My thoughts now?
In retrospect, we should have paid more attention to the normative aspects of recommendation and advertising systems. What do we (platform, content producers, content consumers) want from them? What are the long term consequences on network structure? How do you encourage good content? Good interaction? And what does "good" even mean in this context? I am glad that computational social science, and my own research, has evolved so that these kinds of epistemic questions are now being seriously studied.