.svg)
“Hypergrowth and exponential stuff seems fun to talk about when you look at the numbers and the graphs, and the hockey sticks. But when you have to live it and actually make it happen, it is excruciating painful.”
Thuan Pham doesn’t mince words about his time as CTO of Uber: Meteoric growth is hard. For Uber, growth in the early days was practically exponential. They were adding markets a dozen at a time, rolling out new services, testing features, all on top of an incredibly complicated marketplace. What’s a simple tap to request a ride experience for the consumer requires systems for matching rider to driver, routing rides, determining dynamic pricing, and processing payments, paying drivers, and keeping the entire experience stable and secure.
In Thuan’s Equivalent to Magic podcast interview, he shares that there’s no avoiding having to make trade offs when you’re scaling as fast as Uber did in its early years. And, that no matter what, there will be massive failures from time to time. What counts is how you lead through and learn from those most challenging moments.
Listen in now to Thuan’s episode on Equivalent to Magic or check out the excerpt below from his conversation with Steve Herrod and Quentin Clark.
Thuan Pham | Equivalent to Magic
Quentin Clark: Thuan started at Uber in 2013 when ride sharing was just in a couple of dozen cities, and when he left in May of 2020, Uber was in more than 900 cities and booking billions of rides per quarter. We talked with Thuan how he kept the technical structure of the organization intact as a company ballooned, starting with the moment he arrived.
Thuan Pham: The service was very small. We were only doing 30,000 trips a day, right, in the total Uber system, and Uber was only in around 20 or 30 cities at the time. Even then, the system was not robust enough. It would just break left, right, and center multiple times a week. In my first couple years on the job, I probably would have to wake up in the middle of the night a few times a week actually fighting with all these outages with the engineers.
“…the confluence of all of those things made it an incredibly challenging job and a very interesting job. It’s an experience that I wouldn’t have traded for anything.”
At the same time, the product was starting to catch fire. There was hockey stick growth everywhere. So you have to retrofit there, fix all the broken architectures and systems while you’re having to scale and keep up with the pace of the business and building new things as well, build a new feature, a new capability, all of that stuff, and the confluence of all of those things made it an incredibly challenging job and a very interesting job. It’s an experience that I wouldn’t have traded for anything. That kind of trial by fire is really awesome.
Steve Herrod: You launched Uber X which is seemingly an order of magnitude more complex than the black car service. Tell us what that was like, how quickly did you have to scale? What did you do? People and technology-wise to achieve that one?
Thuan Pham: Launching Uber X was a huge inflection point for us in terms of growth. Back then the black car experience was a beautiful experience, but it’s pretty expensive. There aren’t that many black cars in every city. But the moment that we opened up peer to peer, that’s when we see that every city went through that knee of the curve when you actually bend upward into this hockey stick. Then at the same time, Travis and the ops team were launching more and more cities. So as an engineering team, we had to predict how fast this thing’s going to grow. We have to figure out what system we have to build and rebuild in order to stay ahead of that growth.
Capacity is easy enough that when you design it properly, you can just throw hardware at the problem. But the problem for us is that you have to build a system that actually can tolerate that kind of a scale, or is they actually stay ahead of that kind of a scale.
So the moment you roll out a system, you have to start to redesign the next generation of systems again, right? Because you don’t want to over-engineer and build something that’s 100X or 1,000X, and then it would be too long, and then you don’t survive. The load and stuff will crush you, and you die before you can roll out that system. So you do it just enough to survive, and then you have to figure out how to do it again and again and again.
Quentin Clark: Were there things that you ended up having to scale that you’re surprised ended up becoming interesting scale challenges?
Thuan Pham: The thing that surprised me is under extreme growth conditions like that, how hard things become that would normally seem pretty simple. Let me give you an example. When I came into the company, there were just really three simple services, really. There was a dispatch service that matches riders and driver demand and supply. Then there’s a backend service that basically it’s called the API. It’s a monolith that has all of the business API. If you want to onboard and register a driver, you call the API, and you create a new driver. You do billing and payment. You call this monolith with all these API. We knew that this API model was going to be a problem. So we have to dismantle it into microservices.
If I have an engineering team that can do nothing but tackling that problem, we can probably knock it off in probably two months. But it turned out it took us two years to do that because growth was happening. We have to keep on building new features and new capabilities to help the business grow. Then meanwhile, we did occasionally move one module, one setup API out of that monolith into the set of microservice. But then as soon as you move that out, the remaining stuff blew up in terms of 5X, 10X of that, and it gets more complicated and then new functionality gets added onto that monolith as well.
Quentin Clark:
What are some of the most important things that you did to maintain the right culture and posture and capability as you scale that engineering team?
Thuan Pham:
We did a few things. One is we made sure that the organizational structure has a way to scale so that we can all operate in a loosely coupled but highly aligned manner, while we organize ourselves functionally for the employee career development, meaning engineer report to engineering manager, report to engineering directors. But when we actually worked together, people were organized into product area teams. These teams are cross-functional and work together to solve the same problem. We even assigned a recruiting team to each of these product areas as well.
[With this structure,] we empower everyone to solve their own problem, rather than giving people the excuse to point fingers as some centralized function, like recruiting or even engineering. So we basically said, “You all own that particular area. Go figure out the roadmap. Go figure out the resources, and then you make it happen.” That allows people just able to focus on what they need to do and do it now.
There is no structure that is actually perfect, right? For that, we get a lot in terms of speed and agility and independence. The costs of that is the pile of technical debt we created along the way because we were always thin runway, hair-on-fire all the time. Building new things, launching new services. [The teams would] look at another service that’s close enough to what they have. They rip it off and modified the remaining 20%, launched their own service and rolled it out there.
So they get the product to market incredibly fast. But then we ended up with thousands of microservices… and with half a dozen or so storage stacks because people use whatever they need to use or they know how to use in order to get their product to the market really, really quickly.
“There’s always a trade off to every decision that you make. But in those early years, we had no choice but to structure ourselves so that we can grow.”
After the hypergrowth phase, we had to come back, and we’re still doing that today to basically map that up and standardize things and deprecate things and remove the redundancy in order to gain back their efficiency. So there’s always a trade off to every decision that you make. But in those early years, we had no choice but to structure ourselves so that we can grow because growth was a number one priority for the company.
Equivalent to Magic
Quentin Clark: One of the things that we’re asking all of our guests is if they could describe a single magic moment from their career, a moment where you truly realize the magic of technology.
Thuan Pham: Yeah. How about I’ll give you two, actually. It just so happened they almost perfectly bookend my career so far.
So my second job out of school, I got myself into Silicon Graphics. At the time, SGI was doing interactive TV technology. This is pre-internet as we know today. Netscape wasn’t founded yet, right? The internet at the time was basically just the system that people can email around between school, really. There in that division called the IDS, interactive digital solution division within SGI, we were already building a closed-loop network that has 4,000 homes with TVs and set-top boxes that can already stream movie on-demand, that allowed you to order things online through that closed system, that allowed you to play a game with other people online through your TV set through your set-top.
At the time, it was so magical because these days, we take it all for granted. But back then, there was no internet that people know of; most people didn’t even have email at that time. We were able to create this virtual environment where people have everything on demand like that.
So that was the most magical thing.
The technology was so sexy that all the celebrities in Hollywood came through SGI to see demos, including Michael Jackson and Steven Spielberg and all of these folks. So that was the time of my life.
Fast forward 25, 30 years at Uber, the magical experience now that I’ve done and looking back at it was just a marvel of how quickly we were able to pull all these technologies together and literally transformed the experience of how people transport themselves in a city. Just a magical experience: You just push a button, and the car will get there and just whisk you away.
Within a few short years from 2013 when I joined, we went from just in a couple dozen cities to in 2017, more than 600 cities around the world or something like that. In just a few short years, we were able to deliver that kind of magical experience powered by technology to millions and millions of people around the world.
“Hypergrowth and exponential stuff seems fun to talk about when you look at the numbers and the graphs, and the hockey sticks. But when you have to live it and actually make it happen, it is excruciating painful.”
Thuan Pham doesn’t mince words about his time as CTO of Uber: Meteoric growth is hard. For Uber, growth in the early days was practically exponential. They were adding markets a dozen at a time, rolling out new services, testing features, all on top of an incredibly complicated marketplace. What’s a simple tap to request a ride experience for the consumer requires systems for matching rider to driver, routing rides, determining dynamic pricing, and processing payments, paying drivers, and keeping the entire experience stable and secure.
In Thuan’s Equivalent to Magic podcast interview, he shares that there’s no avoiding having to make trade offs when you’re scaling as fast as Uber did in its early years. And, that no matter what, there will be massive failures from time to time. What counts is how you lead through and learn from those most challenging moments.
Listen in now to Thuan’s episode on Equivalent to Magic or check out the excerpt below from his conversation with Steve Herrod and Quentin Clark.
Thuan Pham | Equivalent to Magic
Quentin Clark: Thuan started at Uber in 2013 when ride sharing was just in a couple of dozen cities, and when he left in May of 2020, Uber was in more than 900 cities and booking billions of rides per quarter. We talked with Thuan how he kept the technical structure of the organization intact as a company ballooned, starting with the moment he arrived.
Thuan Pham: The service was very small. We were only doing 30,000 trips a day, right, in the total Uber system, and Uber was only in around 20 or 30 cities at the time. Even then, the system was not robust enough. It would just break left, right, and center multiple times a week. In my first couple years on the job, I probably would have to wake up in the middle of the night a few times a week actually fighting with all these outages with the engineers.
“…the confluence of all of those things made it an incredibly challenging job and a very interesting job. It’s an experience that I wouldn’t have traded for anything.”
At the same time, the product was starting to catch fire. There was hockey stick growth everywhere. So you have to retrofit there, fix all the broken architectures and systems while you’re having to scale and keep up with the pace of the business and building new things as well, build a new feature, a new capability, all of that stuff, and the confluence of all of those things made it an incredibly challenging job and a very interesting job. It’s an experience that I wouldn’t have traded for anything. That kind of trial by fire is really awesome.
Steve Herrod: You launched Uber X which is seemingly an order of magnitude more complex than the black car service. Tell us what that was like, how quickly did you have to scale? What did you do? People and technology-wise to achieve that one?
Thuan Pham: Launching Uber X was a huge inflection point for us in terms of growth. Back then the black car experience was a beautiful experience, but it’s pretty expensive. There aren’t that many black cars in every city. But the moment that we opened up peer to peer, that’s when we see that every city went through that knee of the curve when you actually bend upward into this hockey stick. Then at the same time, Travis and the ops team were launching more and more cities. So as an engineering team, we had to predict how fast this thing’s going to grow. We have to figure out what system we have to build and rebuild in order to stay ahead of that growth.
Capacity is easy enough that when you design it properly, you can just throw hardware at the problem. But the problem for us is that you have to build a system that actually can tolerate that kind of a scale, or is they actually stay ahead of that kind of a scale.
So the moment you roll out a system, you have to start to redesign the next generation of systems again, right? Because you don’t want to over-engineer and build something that’s 100X or 1,000X, and then it would be too long, and then you don’t survive. The load and stuff will crush you, and you die before you can roll out that system. So you do it just enough to survive, and then you have to figure out how to do it again and again and again.
Quentin Clark: Were there things that you ended up having to scale that you’re surprised ended up becoming interesting scale challenges?
Thuan Pham: The thing that surprised me is under extreme growth conditions like that, how hard things become that would normally seem pretty simple. Let me give you an example. When I came into the company, there were just really three simple services, really. There was a dispatch service that matches riders and driver demand and supply. Then there’s a backend service that basically it’s called the API. It’s a monolith that has all of the business API. If you want to onboard and register a driver, you call the API, and you create a new driver. You do billing and payment. You call this monolith with all these API. We knew that this API model was going to be a problem. So we have to dismantle it into microservices.
If I have an engineering team that can do nothing but tackling that problem, we can probably knock it off in probably two months. But it turned out it took us two years to do that because growth was happening. We have to keep on building new features and new capabilities to help the business grow. Then meanwhile, we did occasionally move one module, one setup API out of that monolith into the set of microservice. But then as soon as you move that out, the remaining stuff blew up in terms of 5X, 10X of that, and it gets more complicated and then new functionality gets added onto that monolith as well.
Quentin Clark:
What are some of the most important things that you did to maintain the right culture and posture and capability as you scale that engineering team?
Thuan Pham:
We did a few things. One is we made sure that the organizational structure has a way to scale so that we can all operate in a loosely coupled but highly aligned manner, while we organize ourselves functionally for the employee career development, meaning engineer report to engineering manager, report to engineering directors. But when we actually worked together, people were organized into product area teams. These teams are cross-functional and work together to solve the same problem. We even assigned a recruiting team to each of these product areas as well.
[With this structure,] we empower everyone to solve their own problem, rather than giving people the excuse to point fingers as some centralized function, like recruiting or even engineering. So we basically said, “You all own that particular area. Go figure out the roadmap. Go figure out the resources, and then you make it happen.” That allows people just able to focus on what they need to do and do it now.
There is no structure that is actually perfect, right? For that, we get a lot in terms of speed and agility and independence. The costs of that is the pile of technical debt we created along the way because we were always thin runway, hair-on-fire all the time. Building new things, launching new services. [The teams would] look at another service that’s close enough to what they have. They rip it off and modified the remaining 20%, launched their own service and rolled it out there.
So they get the product to market incredibly fast. But then we ended up with thousands of microservices… and with half a dozen or so storage stacks because people use whatever they need to use or they know how to use in order to get their product to the market really, really quickly.
“There’s always a trade off to every decision that you make. But in those early years, we had no choice but to structure ourselves so that we can grow.”
After the hypergrowth phase, we had to come back, and we’re still doing that today to basically map that up and standardize things and deprecate things and remove the redundancy in order to gain back their efficiency. So there’s always a trade off to every decision that you make. But in those early years, we had no choice but to structure ourselves so that we can grow because growth was a number one priority for the company.
Equivalent to Magic
Quentin Clark: One of the things that we’re asking all of our guests is if they could describe a single magic moment from their career, a moment where you truly realize the magic of technology.
Thuan Pham: Yeah. How about I’ll give you two, actually. It just so happened they almost perfectly bookend my career so far.
So my second job out of school, I got myself into Silicon Graphics. At the time, SGI was doing interactive TV technology. This is pre-internet as we know today. Netscape wasn’t founded yet, right? The internet at the time was basically just the system that people can email around between school, really. There in that division called the IDS, interactive digital solution division within SGI, we were already building a closed-loop network that has 4,000 homes with TVs and set-top boxes that can already stream movie on-demand, that allowed you to order things online through that closed system, that allowed you to play a game with other people online through your TV set through your set-top.
At the time, it was so magical because these days, we take it all for granted. But back then, there was no internet that people know of; most people didn’t even have email at that time. We were able to create this virtual environment where people have everything on demand like that.
So that was the most magical thing.
The technology was so sexy that all the celebrities in Hollywood came through SGI to see demos, including Michael Jackson and Steven Spielberg and all of these folks. So that was the time of my life.
Fast forward 25, 30 years at Uber, the magical experience now that I’ve done and looking back at it was just a marvel of how quickly we were able to pull all these technologies together and literally transformed the experience of how people transport themselves in a city. Just a magical experience: You just push a button, and the car will get there and just whisk you away.
Within a few short years from 2013 when I joined, we went from just in a couple dozen cities to in 2017, more than 600 cities around the world or something like that. In just a few short years, we were able to deliver that kind of magical experience powered by technology to millions and millions of people around the world.
