Mastering the Art of Marketing Experimentation @ General Catalyst NYC
.svg)
Captured by Denali Tietjen
Last week, we hosted our latest working session with marketing & growth leaders in NYC. Paul Hennessy, CEO of GC-backed Vroom, led the discussion, which centered around customer-centric marketing and driving a culture of experimentation.
While Hennessy would never say this about himself, he’s a bit of a legend among the marketing community. He joined Vroom this summer following a 16-year stint at The Priceline Group where he served as CEO of Priceline.com and CMO of Booking.com. At Booking.com, Hennessy’s team was literally Google’s biggest customer; the team spent over $2 billion on advertising on Google each year, managed over one billion keywords, and had over 10x the bookings of its next competitor.
Hennessy credits Booking.com’s impressive marketing success to its unique culture of experimentation. “Working at Booking.com felt like working in a college research lab,” he said. “You’d hear people saying ‘hey, I’m running to grab food, can you watch my experiment?”
Experimentation is a grand philosophy, but how does it work in practice? Here are 10 takeaways from our conversation with Hennessy:
1. Experiment at high velocity. “Nine out of ten experiments fail, so fail fast,” Hennessy said. Successful experimentation relies on efficiency, so don’t get distracted by what doesn’t work.
At the management level this requires instilling a culture that embraces both failure and success, so engineers aren’t discouraged when you scratch a feature they’ve been working on for days or even weeks. By de-stigmatizing failure, it becomes less personal.
2. Think big, test small. “The ‘one-in-ten’ success ratio holds true regardless of size,” Hennessy said. So why risk doing big experiments? Before making any macro-changes, validate them at the micro-level. If you have to scope it, it’s too big.
3. Make customers the decision makers. Adopt the mantra that no one knows the answer but the customer. Seniority isn’t necessarily correlated with correctness. “At Booking.com, we used this expression ‘don’t be a hippo,’ which stood for ‘highest-paid-person’s-opinion,” Hennessy said, which authority through seniority. Don’t be a hippo.
4. One size doesn’t fit all. No two customers are the same — each has her own individual preferences and behavior. As marketers, consider these preferences and accommodate your messaging accordingly. At Booking.com, certain adjectives resonated differently depending on a customer’s values. When describing the same hotel lobby, “spacious” had a high conversion rate among American customers, while Japanese customers preferred “beautiful.”
5. Have a comprehensive definition of success. While a change may appear positive on surface level, it’s important to consider any collateral damage that may arise. At Booking.com, they often started measuring impact by studying changes in conversion. But they didn’t stop there. While your conversion numbers might improve, your commensurate call center activity might also rise. To measure this secondary impact, the Booking.com team would measure changes to their operating profit as a company.
6. Be creative in in how you structure your organization. Consider breaking any large teams into multiple smaller ones. This small-team structure enables your company to work on many projects simultaneously without getting too caught up in any particular one.
At Booking.com, this took the form of 175 teams of three people — one designer, one developer, and one business person. Each team member was fluent in the other members’ disciplines, which fostered independent thinking. A crucial test in Booking.com hiring was finding engineers with commercial instincts and customer empathy. You can find stellar developers that lack these traits, but they found that those individuals were less nimble in their customer & testing-centric environment.
7. Adapt testing culture to your sector. Applying this philosophy of experimentation can look different depending on your company’s industry and scope.
In an industry with a short sales cycle like online retail, for example, you can run a different experiment every 24 hours. If you’re in the business of selling cars, however, that sales cycle jumps to 30 days, which limits how often you can experiment.
Another parameter to consider is sales volume. “At Booking.com, we could get statistically significant data in 50 minutes. At Vroom, that number is three weeks. For a smaller startup, that might be two months,” Hennessy explained. “That means we can run ten experiments at a time instead of a hundred.”
8. Rely on behavioral science, not just data science. Not every human emotion can be quantified. If you’re exclusively data-driven, you fail to capture the emotional aspects of consumer behavior.
Recognizing this reality, Hennessy has developers on his teams read books on love and relationships to better understand human psychology. “Our most successful conversion a-ha moments [at Booking.com] came from one of these ‘love books.’”
The book described a common relationship phenomenon: Say you’re working up the nerve to talk to your crush when your friend walks over and says she has a crush on the same person. Suddenly, you become even more interested. Upon reading about this phenomenon, one of Hennessy’s engineers decided to apply it to online bookings. Thus the “x-number of people are looking at this hotel” feature was born and conversion soared. Social proof!
9. Imagine you have an unlimited marketing budget. Marketing budgets appear as a constraint. As a marketer, your job is to identify profitable channels. To the extent that you’ve found them, you should campaign to push the confines of your budget. Wasting money on marketing is one thing. Making money is another.
10. Beware of information overload. There’s a fine line between informative and overwhelming marketing. Focus your messaging on product and experience, and only include information that’s essential to your customer now.
“Now” refers not only to the point in your customer’s buying journey, but how mature your industry is. It’s safe to assume people are familiar with booking hotel reservations online, so Booking.com’s marketing is completely experience-focused. People are less accustomed to buying cars online, however, so Vroom provides more information upfront.
Currently, Vroom’s initial messaging highlights “free-shipping” and a “seven-day test drive.” As more people start buying cars online and the process become more familiar, customers won’t demand this information upfront, Hennessy explained. “The minute we don’t need that message, it’ll come out.”
Captured by Denali Tietjen
Last week, we hosted our latest working session with marketing & growth leaders in NYC. Paul Hennessy, CEO of GC-backed Vroom, led the discussion, which centered around customer-centric marketing and driving a culture of experimentation.
While Hennessy would never say this about himself, he’s a bit of a legend among the marketing community. He joined Vroom this summer following a 16-year stint at The Priceline Group where he served as CEO of Priceline.com and CMO of Booking.com. At Booking.com, Hennessy’s team was literally Google’s biggest customer; the team spent over $2 billion on advertising on Google each year, managed over one billion keywords, and had over 10x the bookings of its next competitor.
Hennessy credits Booking.com’s impressive marketing success to its unique culture of experimentation. “Working at Booking.com felt like working in a college research lab,” he said. “You’d hear people saying ‘hey, I’m running to grab food, can you watch my experiment?”
Experimentation is a grand philosophy, but how does it work in practice? Here are 10 takeaways from our conversation with Hennessy:
1. Experiment at high velocity. “Nine out of ten experiments fail, so fail fast,” Hennessy said. Successful experimentation relies on efficiency, so don’t get distracted by what doesn’t work.
At the management level this requires instilling a culture that embraces both failure and success, so engineers aren’t discouraged when you scratch a feature they’ve been working on for days or even weeks. By de-stigmatizing failure, it becomes less personal.
2. Think big, test small. “The ‘one-in-ten’ success ratio holds true regardless of size,” Hennessy said. So why risk doing big experiments? Before making any macro-changes, validate them at the micro-level. If you have to scope it, it’s too big.
3. Make customers the decision makers. Adopt the mantra that no one knows the answer but the customer. Seniority isn’t necessarily correlated with correctness. “At Booking.com, we used this expression ‘don’t be a hippo,’ which stood for ‘highest-paid-person’s-opinion,” Hennessy said, which authority through seniority. Don’t be a hippo.
4. One size doesn’t fit all. No two customers are the same — each has her own individual preferences and behavior. As marketers, consider these preferences and accommodate your messaging accordingly. At Booking.com, certain adjectives resonated differently depending on a customer’s values. When describing the same hotel lobby, “spacious” had a high conversion rate among American customers, while Japanese customers preferred “beautiful.”
5. Have a comprehensive definition of success. While a change may appear positive on surface level, it’s important to consider any collateral damage that may arise. At Booking.com, they often started measuring impact by studying changes in conversion. But they didn’t stop there. While your conversion numbers might improve, your commensurate call center activity might also rise. To measure this secondary impact, the Booking.com team would measure changes to their operating profit as a company.
6. Be creative in in how you structure your organization. Consider breaking any large teams into multiple smaller ones. This small-team structure enables your company to work on many projects simultaneously without getting too caught up in any particular one.
At Booking.com, this took the form of 175 teams of three people — one designer, one developer, and one business person. Each team member was fluent in the other members’ disciplines, which fostered independent thinking. A crucial test in Booking.com hiring was finding engineers with commercial instincts and customer empathy. You can find stellar developers that lack these traits, but they found that those individuals were less nimble in their customer & testing-centric environment.
7. Adapt testing culture to your sector. Applying this philosophy of experimentation can look different depending on your company’s industry and scope.
In an industry with a short sales cycle like online retail, for example, you can run a different experiment every 24 hours. If you’re in the business of selling cars, however, that sales cycle jumps to 30 days, which limits how often you can experiment.
Another parameter to consider is sales volume. “At Booking.com, we could get statistically significant data in 50 minutes. At Vroom, that number is three weeks. For a smaller startup, that might be two months,” Hennessy explained. “That means we can run ten experiments at a time instead of a hundred.”
8. Rely on behavioral science, not just data science. Not every human emotion can be quantified. If you’re exclusively data-driven, you fail to capture the emotional aspects of consumer behavior.
Recognizing this reality, Hennessy has developers on his teams read books on love and relationships to better understand human psychology. “Our most successful conversion a-ha moments [at Booking.com] came from one of these ‘love books.’”
The book described a common relationship phenomenon: Say you’re working up the nerve to talk to your crush when your friend walks over and says she has a crush on the same person. Suddenly, you become even more interested. Upon reading about this phenomenon, one of Hennessy’s engineers decided to apply it to online bookings. Thus the “x-number of people are looking at this hotel” feature was born and conversion soared. Social proof!
9. Imagine you have an unlimited marketing budget. Marketing budgets appear as a constraint. As a marketer, your job is to identify profitable channels. To the extent that you’ve found them, you should campaign to push the confines of your budget. Wasting money on marketing is one thing. Making money is another.
10. Beware of information overload. There’s a fine line between informative and overwhelming marketing. Focus your messaging on product and experience, and only include information that’s essential to your customer now.
“Now” refers not only to the point in your customer’s buying journey, but how mature your industry is. It’s safe to assume people are familiar with booking hotel reservations online, so Booking.com’s marketing is completely experience-focused. People are less accustomed to buying cars online, however, so Vroom provides more information upfront.
Currently, Vroom’s initial messaging highlights “free-shipping” and a “seven-day test drive.” As more people start buying cars online and the process become more familiar, customers won’t demand this information upfront, Hennessy explained. “The minute we don’t need that message, it’ll come out.”
