My research in computer science and human-computer interaction enables access and mobilizes the crowd worldwide for highly complex and long-term endeavors.
I am a postdoctoral researcher at Stanford University, where I work with Sharad Goel and Amin Saberi in the MS&E department and Michael Bernstein in the CS department. My research interests include human-computer interaction, crowdsourcing, human computation+AI and education. The focus of my work has been to scale traditionally hard and open-ended complex endeavors — research, teaching, time-critical campaigns, data collection, writing — where I design and develop computational models and systems to coordinate and empower large number of people to make long-term substantial contributions. Currently, I am on a quest to scale the AI + human-in-the-loop systems, and make scientific contribution and scientific knowledge more accessible and open.
In past, I spent most of my grad school (three years) working with Michael Bernstein at the Stanford HCI Group, and was advised by James Davis at U.C. Santa Cruz, where I earned my M.S. and Ph.D. degrees in CS. During grad school, I also worked with various groups - Reid Porter at LANL, Andrés Monroy-Hernández and Jaime Teevan at MSR Redmond, Victoria Bellotti at PARC, Monica Lam at MobiSocial, Inc., Michael Muller and Werner Geyer at IBM T.J. Watson Research, and Bill Thies and Ed Cutrell at MSR India. I've also had work experience at Google SoC, OpenStreetMap Foundation, OLPC, and Accenture Tech Labs. My research in crowdsourcing has a range of applications in various disciplines of computer science and HCI, if I can help you or if you'd like to collaborate, please feel free to contact me.
Stanford Crowd Research: Enabling Global Access to Research at Scale (MOOR)
Collaborators: Michael Bernstein, Sharad Goel, Geza Kovacs, Ranjay Krishna, Camelia Simoiu, Imanol Arrieta Ibarra at Stanford University. Michael Wilber, Andreas Veit and Serge Belongie at Cornell Tech. James Davis at University of California, Santa Cruz. Snehalkumar (Neil) Gaikwad at MIT Media Lab. Over 1,500 students worldwide.
The Aspiring Researchers Challenge a.k.a. Stanford Crowd Research is an experiment in massive open online research (MOOR), that explores the possibility of research at scale by connecting an expert with crowd (aspiring researchers). In span of six months, we had more than 1,000 sign ups from people around the world with almost no research experience - more than 90% had never published a paper before, about 25% were female and more than 70% were undergraduates. We developed a weekly structure that helped in coordinating crowd, distribute fair credits, educate and train them about research topics and process. The structure utilized peer review to scale the process, that would include series of research phases like, brainstorming, prototyping, development and user-evaluation. Crowd contributed through engineering, design, paper writing and other core aspects of research.
Through our approach, participants worked on three projects in computer vision, data science and human-computer interaction (HCI) - mentored by professors at Stanford University and UC Santa Cruz. And were able to successfully publish one full paper at ACM UIST 2016, one at ACM CSCW 2017 and three work-in-progress papers at top-tier conferences in computer science - ACM UIST and AAAI HCOMP. Many students have gone on to undergrad and graduate school at MIT, Stanford, UC Berkeley, Carnegie Mellon, Cornell and more. We believe that the project is a first step towards realizing research on a wider scale.
Updates and Impact
- Two crowd-authored, full papers got accepted at ACM UIST 2016, Tokyo, Japan and ACM CSCW 2017, Portland, USA.
- Three crowd-authored, work-in-progress papers got accepted at ACM UIST 2015 and AAAI HCOMP 2015.
- Participants have gone on to undergrad and graduate schools like MIT Media Lab, Stanford, UC Berkeley, Cornell, CMU, University of Minnesota, UC San Diego, UC Santa Cruz, Arizona State University, Georgia Tech and more.
- Two crowd participants from Germany and India offered full time RA positions at Stanford University.
- One paper under preparation for PNAS, one under review for ACM CHI 2017. - The HCI project got into the finals of Knights News Challenge'15 (top 20 of 1,000+).
- The HCI project (Daemo) - to build the next generation crowd marketplace is about to go live - that may provide earning opportunities to thousands of people, and help science through multiple experiments and tasks that can be hosted.
- Daemo crowdsourcing marketplace has been successfully used to create a dataset of 100,000+ question answer pairs on 500+ Wikipedia articles — a paper based on this work recently won the best paper award at EMNLP 2016.
- Global access provided to more than 1,500 people across 6 continents.
Meta publications - about the process
- Vaish, R, Gaikwad, S, Kovacs, G, Veit, A, Krishna, R, Arrieta Ibarra, I, Simoiu, C, Wilber, M, Belongie, S, Goel, S, Davis, J, Bernstein, M. "Crowd Research: Open and Scalable University Laboratories", ACM UIST 2017, Quebec City, Canada.
- Vaish, R, Davis, J, Bernstein, M. “Crowdsourcing the Research Process”, Collective Intelligence 2015, Santa Clara, CA.
- Vaish, R. “Crowdsourcing the Research Process”, Doctoral Consortium at AAAI HCOMP 2014, Pittsburgh, PA.
Crowd publications - about the projects crowd researchers worked on
- HCI project - poster paper: Stanford Crowd Research Collective., Vaish, R., Bernstein, M. "Prototype Tasks: Improving Crowdsourcing Results through Rapid, Iterative Task Design". AAAI HCOMP 17, Quebec City, Canada.
- HCI project - full paper: Stanford Crowd Research Collective., Vaish, R., Bernstein, M. "Designing a Constitution for a Self-Governing Crowdsourcing Marketplace ". Collective Intelligence 17, New York, USA.
- HCI project - demo paper: Stanford Crowd Research Collective., Vaish, R., Bernstein, M. "The Daemo Crowdsourcing Marketplace". ACM CSCW 17, Portland, USA.
- HCI project - full paper: Stanford Crowd Research Collective., Vaish, R., Bernstein, M. "Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms". ACM CSCW 17, Portland, USA. (28 co-authors)
- HCI project - full paper: Stanford Crowd Research Collective., Vaish, R., Bernstein, M. "Boomerang: Aligning Worker and Requester Incentives on Crowdsourcing Platforms". ACM UIST 16, Tokyo, Japan. (38 co-authors)
- HCI project: Stanford Crowd Research Collective, Vaish, R, Bernstein, M. “Daemo: a Self-Governed Crowdsourcing Marketplace”, ACM UIST 15, Charlotte, NC. (61 crowd authors)
- Data Science project: Mysore, A. S., .. Vaish, R,.. et al. “Investigating the ‘Wisdom of Crowds’ at Scale”, ACM UIST 2015, Charlotte, NC. (58 crowd authors)
- Computer Vision project: Veit, A., Wilber, M., Vaish, R., Belongie, B., Davis, J., et al. “On Optimizing Human-Machine Task Assignments”. AAAI HCOMP 2015, San Diego, CA. (49 crowd authors)
- Schuster, C, Zhang, B, Vaish, R, Thomas, J, Gomes, P, Davis, J. “RTI Compression for Mobile Devices”, IEEE ICIMu 2014, Kuala Lumpur, Malaysia (Pilot study)
Talk, Posters and Press
- [Poster] Crowd Research: HCI+Design Open House for CHI 2016, Stanford University, 5/8/16
- [Poster] Daemo: HCI+Design Open House for CHI 2016, Stanford University, 5/8/16
- [Talk] LeadGenius, Berkeley, CA, 3/1/16
- [Talk] Berkeley Institute of Design Seminar, UC Berkeley, 2/9/16
- [Talks] Multiple mentions in talks by Michael Bernstein at Stanford, CMU, UIUC, MIT Media Lab and elsewhere.
- [Talk and Poster] SRC/ISSDM Symposium with Los Alamos National Lab at UCSC, 10/15
- [Press] The Tragedy of the Digital Commons, The Atlantic, 6/8/15
- [Talk] Stanford University HCI lunch, 5/27/15
- [Press] The Aspiring Researcher Challenge: An Experiment In Massive Open Online Research. UCSC SOE, 5/20/15
- [Talk] Stanford Women in Computer Science’s ‘eCSpress yourself’, 5/15
- "Thank you for taking on this ambitious project and for letting me be a part of it. I feel privileged. I feel like this group is now a part of my family"
- "This initiative was the best thing that i ever came across as a student not everybody get such a good opportunity to work"
- "This is simply amazing work. I am really glad that I can be part of it. I see how hard it is to be all in different place, time zone and yet solve the same problem. I enjoyed seeing the system go from Zero to Hero"
- "Hats off, Rajan. You did a wonderful job managing the whole thing. Coordination was just great. I'm particularly grateful for your (team's) effort cuz I learnt stuff, first hand, that I might not have had the opportunity to at an undergrad level"
- "Thank you to professor Bernstein and Rajan for letting a high schooler join the project. Both of them are very kind and I enjoy working with them"
Stanford Scholar Initiative: Collaborative Video Production at Scale - Research Talks and Technical Topics
Collaborators: Sharad Goel, Amin Saberi at Stanford University. Over 800 students worldwide.
There are millions of research papers that go unread, some are behind paywall, while some are hard for non-researchers to read and understand. Besides writing papers, researchers lack time and resources to make their work available in other forms or formats. Meanwhile, there's been a shift in consuming information and knowledge through videos. Through this initiative, we're trying to make world's research more accessible - by mobilizing people worldwide to scale the traditional video creation process, and collaboratively create short talks on influential papers - that can be improved/edited by anyone. To accomplish this goal, we're designing a scaffolding process to coordinate the crowd, and building a system to enable collaborative and editable video production. Beyond research talks, we're applying our techniques to develop courses. Eventually, this research can help open-ended media collaboration - from education to journalism, and beyond.
Updates and Publication
- [Paper] Vaish, R., Goel, S., Saberi, A. "Mobilizing the Crowd To Create an Open Repository of Research Talks", ACM Learning@Scale 2017 (Work-in-Progress), MIT, Cambridge, MA.
- [Talk] Stanford Postdoc Research Symposium, 12/6/2016.
- [Numbers] 800+ strong community, 11,000+ views, 140+ videos, 24,000+ minutes of watch time.
- [Progress] 8 research talks completed, based on best papers at WWW 2015 and WWW 2016.
- [Progress] 4 short courses completed, in multiple languages.
- [Talk] Social Computing Lab, Stanford University, 5/23/16.
- "I had always wanted to read through research papers on hot topics of computer science. But I could never get started. This program not only inspires me to read through a paper but requires me to understand it enough, so as to create a talk on it. And learning is always fun when more people are learning with us." - from a student from India.
- "Thanks again for your efforts on our behalf. You and your team are providing a great public service to science." - from a professor from Kellogg School of Management, U. Penn.
- "The talk is extremely thorough despite its brevity. Its certainly better than the talk I gave at WWW." - from a researcher at Wikimedia Research, San Francisco, CA.
- "Thank you so much for the reference and opportunity to join this initiative Rajan. I’ve been taking courses for so long but being able to create my own and help out with education was very special to me." - a high school student who recently got accepted to Stanford's class of 2021.
The Whodunit Challenge: Mobilizing the Crowd in India
Collaborators: Bill Thies and Ed Cutrell at Microsoft Research India. Aditya Vashistha at University of Washington.
While there has been a surge of interest in mobilizing the crowd to solve large-scale time-critical challenges, to date such work has focused on high income countries and Internet-based solutions. In developing countries, approaches for crowd mobilization are often broader and more diverse, utilizing not only the Internet but also face-to-face and mobile communications. The Whodunit Challenge is first of its kind social mobilization contest to be launched in India. The contest enabled participation via basic mobile phones and required rapid formation of large teams in order to solve a fictional mystery case. The challenge encompassed 7,700 participants in a single day and was won by a university team in about 5 hours. To understand teams strategies and experiences, we conducted 84 phone interviews. While the Internet was an important tool for most teams, in contrast to prior challenges we also found heavy reliance on personal networks and offline communication channels.
- Vashistha, A, Vaish, R, Cutrell, E, Thies, W; “The Whodunit Challenge: Mobilizing the Crowd in India”; INTERACT 2015, Bamberg, Germany. (Equal contributions by Vashistha and Vaish).
Press - sadly, few links are not live anymore
- Microsoft to test social tech in India, Times of India, 2/13
- Microsoft’s social ‘Whodunit’ competition to begin in India, Yahoo! News, 2/13
- From Computing Research to Surprising Inventions (Peter Lee, head of MSR, launching the Whodunit? Challenge at TechFest India), Microsoft Research, 2/13
- Microsoft India Announces A Nationwide Social Gaming Competition, Silicon India, 1/13
- Social whodunnit competition launches in India, NewScientist, 1/13
- More press by Business Standard, CNN IBN, ACM.org, IIIT-Delhi and more...
Twitch Crowdsourcing: Crowd Contributions in Short Bursts of Time
Collaborators: Michael Bernstein, Keith Wyngarden, Jingshu Chen, Brandon Cheung at Stanford University.
To lower the threshold to participation in crowdsourcing, we present twitch crowdsourcing: crowdsourcing via quick contributions that can be completed in one or two seconds. We introduce Twitch, a mobile phone application that asks users to make a micro-contribution each time they unlock their phone. Twitch takes advantage of the common habit of turning to the mobile phone in spare moments. Twitch crowdsourcing activities span goals such as authoring a census of local human activity, rating stock photos, and extracting structured data from Wikipedia pages. At the time of CHI’14 paper submission, 82 users made 11,240 crowdsourcing contributions as they used their phone in the course of everyday life. After six months of deployment, over 100,000 contributions were registered. The median Twitch activity took just 1.6 seconds, incurring no statistically distinguishable costs to unlock speed or cognitive load compared to a standard slide-to-unlock interface.
- Vaish, R, Wyngarden, K, Cheung, B, Bernstein, M. “Twitch Crowdsourcing: Crowd Contributions in Short Bursts of Time”, CHI’14, Toronto, Canada.
Talks, Posters, Press and Achievements
- [Talk] Stanford University HCI lunch, 6/19/13
- [Talk] MIT CSAIL, 7/16/13
- [Talk and poster] Stanford MobiSocial Retreat, 10/5/13 http://mobisocial.stanford.edu/retreat13/
- [Poster] UC Santa Cruz Research Review Day, 10/17/13
- [Press] What’s new in digital and social media research: Crowdsourcing, analytics, Twitter patterns, product ratings, Harvard’s Journalist’s Resource, 5/14
- [Press] The Next Frontier in Crowdsourcing: Your Smartphone, MIT Technology Review, 3/14
- [Press] Crowdswiping, Stanford The Dish Daily, 2/14
- [Press] Crowdsourcing with a swipe of your finger, Santa Cruz Sentinal/San Jose Mercury, 2/14
- [Press] Crowdsourcing Twitch app could turn swipes into cash, NewScientist, 1/14
- [Achievement] Michael Bernstein was awarded the Google Faculty Grant for Twitch crowdsourcing proposal.
Exploring employment opportunities through microtasks via cybercafés
Collaborators: James Davis at U.C. Santa Cruz. Mrunal Gawade at Centrum Wiskunde & Informatica, Amsterdam.
Microwork in cybercafés is a promising tool for poverty alleviation. For those who cannot afford a computer, cybercafés can serve as a simple payment channel and as a platform to work. However, there are questions about whether workers are interested in working in cybercafés, whether cybercafé owners are willing to host such a set up, and whether workers are skilled enough to earn an acceptable pay rate? We designed experiments in internet/cyber cafes in India and Kenya to investigate these issues. We also investigated whether computers make workers more productive than mobile platforms? In surveys, we found that 99% of the users wanted to continue with the experiment in cybercafé, while 8 of 9 cybercafé owners showed interest to host this experiment. User typing speed was adequate to earn a pay rate comparable to their existing wages, and the fastest workers were approximately twice as productive using a computer platform.
- Gawade, M, Vaish, R, Waihumbu, M. N, Davis, J; “Exploring Employment Opportunities through Microtasks via Cybercafés”; IEEE Global Humanitarian Technology Conference 2012, Seattle, WA.
- Gawade, M, Vaish, R, Waihumbu, M. N, Davis, J; “Exploring microwork opportunities through cybercafes”; ACM DEV 2012, Atlanta, GA. Work-in-progress.
Poster and Achievements
- [Achievement] Semi-finalist at the UC Berkeley Global Social Venture Competition 2012.
- [Poster] UC Berkeley CITRIS Retreat 2013, Berkeley, CA.
Internet vs. Enterprise Crowdfunding: Contrasting Motivations and Dynamics
Collaborators: Michael Muller, Werner Geyer and Todd Soule at IBM T.J. Watson Research, Cambridge, MA.
In this project we contrast crowdfunding as it occurs on the Internet, with crowdfunding in an Enterprise setting, based on a grounded theory analysis of a crowdfunding trial at IBM Research Almaden. We explore themes of diverse projects, motivations and incentives, strategies and approaches, and collaborations and relationships. Enterprise crowdfunding has its own financial model, social scope, and dynamics, resulting from a heightened sense of collaboration and community. This project helps us learn about the implications for organizations and for future crowdfunding activities.
- Vaish, R, Muller, M, Geyer, W, Soule, T. “Crowdfunding in the Enterprise and on the Internet: Workplace Users Emphasize Collaboration and Sociality”, Research Report RC25535, IBM Research 2015.
Talk, Poster and Press
- [Talk and Poster] InternFest’13, IBM Research, Cambridge, MA.
- [Press] IBM discovers its inner Kickstarter via enterprise crowdfunding, Network World, 8/13.
- [Press] Can Internal Crowdfunding Help Companies Surface Their Best Ideas?, Harvard Business Review, 9/13.
Peerworthy: Motivating Participation in Prosocial Peer-to-Peer Services
Collaborators: Victoria Bellotti at Palo Alto Research Center (PARC). Vera Liao at Univ. at Illinois Urbana-Champaign.
Internet is embracing peer-to-peer services, and the "sharing economy" they create. This project aims at exploring how to motivate people to join prosocial p2p services that do not involve monetary rewards. We carried out qualitative studies with users of existing online P2P services and offline peer-production groups to uncover A wide range of motivations. We conducted a field experiment with a P2P curating platform we created to compare user recruiting strategies that leverage these motivatiors. A full paper is under submission at International Journal of Human-Computer Studies.
- Vaish, R, Liao, Q. V., Bellotti, V. "What’s in it for me? Self-serving versus other-oriented framing in messages advocating use of prosocial peer-to-peer services", Elsevier International Journal of Human-Computer Studies V109.
Talk and Poster
- [Talk] Palo Alto Research Center, 10/14
- [Poster] Palo Alto Research Center, 7/14
- Publication under review for Elsevier International Journal of Human-Computer Studies.
Crowd-Powered Tone Improvement System for Emails
Collaborators: Andrés Monroy-Hernández and Jaime Teevan at Microsoft Research Redmond.
Communicating a message with right tone is extremely important, misinterpretation of which can cause confusion or unexpected reaction and response. However, people often fail to express the right tone in their messages due to a lack of skills or the wrong assumptions. Can we help them? There are several automated solutions to detect the tone and identify shortcomings, like IBM's Tone Analyzer and ToneCheck. However, their accuracy and reliability if often doubtful. Also, these applications can give recommendations, but cannot fix the tone of the message or text. As part of our project, our system relies on crowd intelligence to improve the tone of emails. The backend of our system is powered by MTurk, where crowd follows a systematically designed and scaffolded workflow. The system inputs email content and context, and outputs improved email. Based on a study on 29 emails, we found that more than 90\% of the emails went through some or significant improvements.
Talk and Publication
- [Talk] Microsoft Research Redmond, 9/15
- [Paper] Vaish, R, Monroy-Hernández, A. "CrowdTone: Crowd-powered tone feedback and improvement system for emails", MSR-TR-2017-1. Also, available on ArXiV.
Other projects from my grad school at U.C. Santa Cruz
3D+2D TV: 3D Displays with No Ghosting for Viewers Without Glasses
3D displays are increasingly popular in consumer and commercial applications. Many such displays show 3D images to viewers wearing special glasses, while showing an incomprehensible double image to viewers without glasses. We demonstrate a simple method that provides those with glasses 3D experience, while viewers without glasses see a 2D image without artifacts.
3D+2D TV project home site, SIGGRAPH site
Digitization of Health Records in Rural Villages- with PAMF
In this project, we present a study that reviews current available methods for obtaining electronic health records (EHRs) to facilitate the provision of health services to patients from rural villages in developing countries. The study compares processes of digitizing health records by means of manual transcription, both by hiring a professional transcriptionist and by using online crowdsourcing platforms. Finally, a cost-benefit analysis is conducted to compare the studied transcription methods to an alternate technology-based solution that was developed to support in-the-field direct data entry.
Using crowdsourcing to generate ground truth data for computer vision training
This project was conducted in collaboration with LANL. Computer vision is great, but at times it fails too. To train the algorithms, usually researchers spend several hours annotating images to create ground truths. Why not harness the crowd here? That’s what we’re trying to do, several experiments have been crowdsourced on Mechanical Turk and MobileWorks. The experiment was conducted on two types of data sets, namely: Pedestrians and Bumble-bees.
Low Effort Crowdsourcing: Leveraging Peripheral Attention for Crowd Work - at CrowdCamp at HCOMP'13, with Jeff Bigham and Haoqi Zhang.
Crowdsourcing systems leverage short bursts of focused attention from many contributors to achieve a goal. By requiring people’s full attention, existing crowdsourcing systems fail to leverage people’s cognitive surplus in the many settings for which they may be distracted, performing or waiting to perform another task, or barely paying attention. In this project, we study opportunities for loweffort crowdsourcing that enable people to contribute to problem solving in such settings. We discuss the design space for low-effort crowdsourcing, and through a series of prototypes, demonstrate interaction techniques, mechanisms, and emerging principles for enabling low-effort crowdsourcing
Before grad school: Older projects and stuff - Class projects, OLPC project, OpenStreetMap project, Yahoo! Open Hack, NASA WorldWind add-on, AOL/Truveo Google Gadges, Microsoft Imagine Cup etc.
Publications and Patents
- Organisciak, P, Vaish, R. "Accomplishing low-attention microtasks", Productivity Decomposed: Getting Big Things Done with Little Microtasks, ACM CHI Workshop 2016, San Jose, CA.
- Scher, S, Liu, J, Vaish, R, Gunawardane, P, Davis, J. “3D+2DTV: 3D Displays with No Ghosting for Viewers without Glasses”, ACM Transaction on Graphics (TOG) 2012.
- Vaish, R, Ishikawa, S, Liu, J, Berkey, S, Strong, P, Davis, J. “Digitization of Health Records in Rural Villages”, IEEE Global Humanitarian Technology Conference 2013, San Jose, CA.
- Vaish, R, Ishikawa, S, Lundquist, S, Porter, R, Davis, J. “Human Computation for Object Detection”, Tech Report UCSC-SOE-15-03, School of Engineering, University of California Santa Cruz.
- Vaish, R, Organisciak, P, Hara, K, Bigham, J, Zhang, H. “Low-effort Crowdsourcing: leveraging peripheral attention for crowd work”, AAAI HCOMP 2014, Pittsburgh, PA [Work-in-progress and Demo].
- James Davis, Steven Scher, Jing Liu, Rajan Vaish, Prabath Gunawardane, "Simultaneous 2D and 3D Images on a Display”, US Patent 2013032821; 2013.
- 3D+2D TV: A 3D display that’s watchable without glasses, without ghosting, Extreme Tech, 6/13.
- UCSC Researchers Develop Display for Both 3D, 2D Viewing, India West, 10/13
- 3D TV faces uncertain future, MSN, 8/13.
- Researchers Develop Ghost-Free 3D For Viewers Not Wearing Glasses, Gizmodo, 7/13.
- UCSC Researchers Develop 3D Display With No Ghosting for Viewers Without Glasses, ACM Comm., 7/13.