The move to microservices was one key to Uber’s growth. But it also came with costs; namely, redundancies and complexities that might not have seeped into a system that was managed in a more top-down or centralized fashion.

By 2018, Uber’s software was a matrix of more than 3,000 microservices.N Many of these services, though, performed overlapping functions. For example, Uber offered promotions to entice drivers to drive. The company had promotions to incentivize riders to ride. There were more promotions for restaurateurs, and more yet for diners using UberEats.

If another Uber team wanted to create an incentive system for another product or customer segment, it was not unusual for them to take some code from the other incentive teams as a baseline, then modify that for their own purposes. The result, said Pham, was “you've got five or six systems that do incentives that are 75 percent similar to one another.”

“It would be fine if all six [systems] were amazing. It’s okay to have six amazing cars in your driveway. But if you have six 30-year-old cars, and one is polluting badly, one’s brakes are broken down and stuff, it’s not useful. You would rather have one car that works right.”

— Ganesh Srinivasan

In October 2017, Pham tasked Srinivasan to corral these close cousins by developing a “Product Platform” — essentially a set of common tools for engineers and developers to do their work. Besides improving efficiency and reducing duplication, the goals included reducing security vulnerabilities and UI inconsistencies. Six hundred people were assigned to Product Platform.

Click here to read an Uber Engineering blog post on Product Platform

Pham also challenged Mengerink and other senior leaders in the engineering organization to clean up Uber’s core infrastructure, such as unifying the compute platform, simplifying the storage stack, and reducing the number of “line [network] protocols” required for all the microservices to interoperate with one another. Pham believed there should be just one or two line protocols, but over time, Uber had developed about a dozen.

“That happens because all of these hundreds of teams make their own independent decisions without checking with one another,” said Pham. “Because they have to go really fast and get things done. They did whatever they felt was workable for them, and they did that with the best of intentions. But as a result, we have this haphazard web of things that are not uniform or standardized. So, the system becomes clunky and hard to manage.”

Convincing people to focus on tasks like reducing the number of line protocols, he admitted, wasn’t easy.

 “Doing this back-end work takes time, and is the kind of work that is unsexy,” said Pham. “It doesn’t really shift the functionality. You're just making it more efficient. It's very hard to get an engineer to do that kind of work; work that is not visible. So, that is a challenge we have.”  

Product Platform wasn’t the only new organizational structure put in place to address technical debt. Pham also backed the creation of Project Ark, which was the brainchild of M. Waleed Kadous, engineering strategy lead for the Office of the CTO at Uber. N

Hatched in the summer of 2017, Project Ark brought together about 20 senior representatives of various “organizations” within the Engineering division — for example, UberEats, Maps, Uber Freight, Storage, Data, Finance and more. The idea was to convene a representative body to tackle issues of technical debt. Kadous recalled:

I had started to see a lot of inefficiency. I would compare … what I thought would be reasonable at my previous employer, Google, and my engineers [at Uber] were only capable of maybe 60 percent or 70 percent of what an equivalent engineer at Google would do. Not because they were any less talented. The talent levels are more or less the same. But it was because the systems and tools and approaches and methodology that Google had built … had really empowered engineers to be efficient. Get stuff out the door without doing a lot of what we call "yak shaving."  That means you've got to do these weird contraptions and weird steps. You want to put a service out and there's this document that has 13 steps in it and running this script and doing this and that. It was very reliant on organizational memory. The systems weren't elegant to use. There were frequently issues that were unexpected.

In September 2017, the Project Ark members got together for a day and decided on six major areas to focus on:

1. Engineer productivity: Engineers spent a lot of time working through unnecessarily complex processes and dealing with unstable systems. 

2. Engineer alignment across teams: Teams often did things in different ways and when it came time to connect them things didn't work out. If they talked first, things would run more smoothly. 

3. Knowledge access and documentation: It was hard to find out how to do things, and it was sometimes hard to even find out who was responsible for a service. 

4. Duplication: There was often more than one way to do a thing at Uber.

5. Unmaintained critical systems: There were some key components at Uber that everyone relied on that were maintained on more or less a volunteer basis. 

6. Culture: Uber’s “Let Builders Build" culture prioritized shipping stuff out the door with little thought to long-term maintainability. We needed to address that. N

Project Ark members thought about their task in terms of three time horizons: quick wins; things that could be tackled in time to have meaningful input to the 2018 planning cycle, and long-term issues.

Quick wins would be important, said Kadous, in part because there was skepticism around the Ark effort. A previous cross-team project known as Panama, he said, “had mixed success.”

Kadous sought to understand why Panama had fallen short. One reason, he said, was the group was not representative of all stakeholders, so getting buy-in on its decisions was difficult. Another reason was that there was little follow-through. 

“Once a decision was made – like, for example, ‘these are the standard set of languages that should be used at Uber’ – nobody followed through to actually make sure that the set of languages we practically used was reduced,” he said of Panama.

Project Ark team members met once a week; there was also an “Ark Admin” group of three to four people who met three times a week to check on the progress of things. Asked why the project was christened “Ark,” Kadous said with a laugh: “I think it was because I felt we were sinking. It was meant to help us get to safer ground. I think maybe that’s the best way to put it.”

Among the initiatives Project Ark launched: eliminating the programming languages Python and JavaScript from Uber products, and setting up a three-year plan to reduce the number of code repositories from 12,000 down to four, and ultimately to one.

Not everyone was excited about Project Ark, admitted Kadous.

M. Waleed Kadous

“Uber had built a culture of ‘Let Builders Build’ and ‘do whatever you want to get the job done,’ and we were coming in and saying, ‘no, for the greater good you're going to have to give up some of your autonomy.’ And not all people were happy with that. And so, there is resistance to change.” 

Kadous also struggled with the fact that Project Ark was, essentially, a volunteer effort. Some participants were more involved than others. 

“The problem with a representative structure is that people have their own organizations, and the dynamics are such that they get pulled back to their primary responsibilities,” he said. “And so, there's always been a struggle to work out how to keep people engaged. Of those 20 representatives, there were six or seven that actually were really, really committed and the other 13 were kind of on the margins. This is still a problem I haven't solved completely, and it's one of the problems of a representative body. You can either choose to be representative, or choose to have a committed group. It's really hard to do both at the same time.” 

Slowly, though, Project Ark started to make a difference.