‘As a developer this is a dream’: There is basically no limit in how hard you can push yourself.
Nothing comes for free. This was a lesson Pham had learned over his two decades in Silicon Valley.
Through his career, Pham had become well acquainted with a concept called technical debt. As Uber scaled up, he knew its engineers and developers needed to move quickly to grow the business and fend off competitors. But move too fast, or too carelessly, and Uber could find itself with a knot of sticky technical problems that could stop it dead in its tracks.
Pham explained:
Just like consumers periodically check their credit card and loan balances, Pham and his team believed it was important to monitor Uber’s technical debt. Doing so was both an art and a science, and the engineering team had quantitative as well as qualitative measures to try to assess the overall health of their system.
Qualitative measures included surveys and listening to day-to-day complaints. Managers would ask engineers how much of their time was spent writing new code versus wrestling with fixing things or dealing with mundane maintenance issues.
“The best measure of the efficacy of development is just the subjective point of view of the developer saying, ‘I’m more effective here than everywhere else I’ve been.’ So you ask a very simple question: How much time do you waste each week?”
--Matthew Mengerink, vice president of core infrastructure engineering.N
Pham and his managers had to distinguish “recreational bitching” from real pain. “Whenever I hear anything from my direct reports, I know I need to move on it immediately, because people on my team have a strong threshold for pain,” said Pham.
Among the quantitative measures were outages and the number of code changes. But even this was tricky — a low number of code changes, for example, could be a good sign (the system is working well and is self-sustaining) or a bad one (the system is so difficult and brittle that people don’t want to touch it).
“There are two clear signals you generally see in all kinds of tech debt situations. Your speed of innovation or development slows down,” said Srinivasan. “Teams will come and ad hoc complain to you. … ‘We want to launch this feature, but the person, team, X is blocking us.’ That’s how these things all start. The second part of it, which is more easily measurable, is just outages, failures. You will just start seeing more instances of failure and you will see more repeatable instances of failure.”
Uber’s engineers and developers were responsible for their own code and on-call for their services, 24/7. That meant one proxy for technical debt was sleep debt.
“If our technical debt is high, and a service is not reliable, if you’re on-call, you’re woken up in the middle of the night 15 times a week,” said Pham. “We have seen some attrition because people say, ‘Actually I didn’t come here for this. I did not sign up for this. I came here to build software and instead I’m mired in this grunt work that I don’t like. I’m going to go to Google because they have beautiful tooling. So there are huge implications.”
The drawback with many of these measurements, noted Srinivasan, was that they were retrospective, rather than prospective. “Generally,” he said, “These are lagging indicators.”
Technical Debt wasn’t the only challenge that would face Uber’s engineering team as the company grew. As Uber scaled rapidly, it hired at a blistering pace. Pham and other managers had to figure out whom to hire, how much to pay them, and whom to promote. How could they develop systems to communicate with, monitor, mentor and coach their rapidly ballooning ranks? How could they establish norms and expectations that inspired and motivated employees?
As with Technical Debt, Pham and his team developed both qualitative and quantitative tools to try to measure Organizational and Cultural Debt. Employees, for example, were surveyed informally and formally about their satisfaction with their managers and the company as a whole.
On formal surveys, employees responded to questions such as “How much do you trust your manager?” “How secure do you feel in your job?” “How much teamwork is there in your group?” “How proud are you to work at Uber?”
Questions were divided into two general categories — company variables and manager variables. Company variables were things like policies and business practices that affected how people feel about their jobs, explained Pham. Manager variables related to how managers took care of their reports. Pham focused more on the latter, which he had greater control over.
“What we can do something about — as [an engineering] organization, as a team — is just to have every manager be obsessive about taking care of their people, especially during the firestorm,” he said. “That built a lot of trust, because when things are going haywire, and engineers are freaking out, and their managers are there taking care of them, listening to their feedback, implementing policies to drive improvements. Employees actually see that and they end up trusting their manager and their team a lot.”
As the engineering organization grew to hundreds of managers, Pham needed a dashboard of sorts to quickly identify potential trouble spots. He would compile the scores for each question for each manager, and color-code them on a spreadsheet. Green boxes signified a high score, white was neutral, and red boxes signified low scores. This allowed him to quickly scan and see which teams might be struggling and need more attention.
The data factored into decisions around promotion and retention. Looking at his spreadsheet and pointing to one supervisor with a string of red boxes, Pham recalled:
Without this data, the VP of this person thinks this person is a very impactful manager, because he's actually in a very important role. Then this person got an offer from another company. He came to us and he said, “I’m about to go to that company. For you to keep me you have to up my compensation.” The VP was like, “Oh, we need to retain that person because he plays such an important role for me.” But I looked at the data and I said, “He sucks. You'll want to churn him out; you'll want to get a better manager. …” And so, we made no attempt to retain him. With this kind of data, you can actually make a much better decision.
Making quantified dashboards from employee surveys wasn’t the only way Pham sought to use data. He made spreadsheets and charts to track the overall level of experience of his rapidly growing division. He also looked at data on attrition and internal transfers, which could reflect Technical as well as Organizational and Cultural Debt, because engineers often leave unhealthy teams to join better ones.
As 2016 progressed, Srinivasan said, signs were mounting that Uber’s debt levels were approaching dangerous territory.
“I think the end of 2016 is when we realized, internally, organically, that the company just cannot scale [any further]. … Both organizationally and from a tech debt perspective, we genuinely can’t build a company. It’s almost like escape velocity when you’re in debt. At some point, you know that you will never really pay your credit card down, and you’ll hit bankruptcy unless you can start doing something.”
-- Ganesh Srinivasan