2.5 Iterating on Previous Growth Experiments

Iterative Experimentation is Key to Growth

This post is a part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

A key part of the growth science methodology is iteration. It’s not enough to form one hypothesis, test it, and implement a change. If you stop short, you’ll be missing out on substantial optimizations that can add up to incredible gains to customer or revenue growth. The key is to keep experimenting in areas where previous changes have led to strategically significant gains.

You can either iterate on past experiments and try small variations or try the same kinds of experiments in other contexts. For example, let’s say that you try an experiment to get more people to click on the registration button on your website. In trying to increase the conversion rate on new customer registration on your web app, you don’t want to stop with just changing the color of the “Register” button to bright green. You also should double down on this line of thinking and test changing the button color to orange and red. Not only that, you might try very related experiments such as testing the label text. Perhaps you can try “Sign up” and “Get Started” instead of “Register.” And these iterations are only for one button. Not only that we haven’t even started talking about marketing and operations.

A great example of iterative experimentation leading to markedly higher increases in growth comes from a leading language learning platform, Duolingo. Gina Gotthilf, VP of Growth, recounted the story of how her team made huge strides in Duolingo’s customer engagement through iterative experimentation with their registration process.

Read more

1.1 Growth Science Pillar 1: Iterative Experimentation

Meme: Never stop testing, and your growth will never stop improving.

The origins of growth science could be traced back to David Ogilvy, one of the most well known ad men of the twentieth century, who proclaimed back in 1963, “Never stop testing, and your advertising will never stop improving.” Ogilvy’s statement embodies two important concepts–continually test and improve–that have paved the way to modern growth science.

The idea that advertising is not a static creation is critically important because challenges one to continuously improve tactics. Although this philosophy was popularized in advertising it is just as relevant in product development, operations, and even sales. Since the 1960’s some marketers have embraced this ethos by iteratively trying variations of tactics, channels and messaging. Moreover, the practice of iteratively implementing improvements has become more commonplace in other business functions such as product development, customer support, and sales. However, It’s not enough to keep trying different tactics and implementations.

The second key concept that Ogilvy proposed is actually testing new variations and improvements by carefully measuring associated outcomes. Once again, this practice is not just limited to marketing activities, and one can test variations of product implementation, operations, and sales. It’s not surprising that many business leaders have adopted both iterative experimentation and rigorous testing across business functions, and it’s now just as common to test variations of product design (particularly in the digital realm) as it is to test different advertising messages.

This paradigm shift has been supported by an ever burgeoning ecosystem of data collection and analysis tools that have allowed companies to track key performance indicators for new customer acquisition, such as click-through rates on online and email marketing, as well as for engagement and retention of existing customers. Now, one can easily test variations to marketing emails or to webpage designs to see which ones lead to the greatest gains in key metrics such as the total number of new sign ups or daily active users.

What was once driven entirely by intuition can now be informed by experimentation and testing. Leaders across marketing, product development, and operations are forming hypotheses and conducting experiments to see which tactics yield the most positive results. This rigorous methodology is the first pillar of growth science and underlies the entire field allowing us to call it a “science.”

What is shocking is that after more than half a century, most marketers still do not make data-driven decisions according to a survey conducted by Google in 2017.  I can empathize with those that do not put this philosophy into practice. While the field has burgeoned, it has also become a lot more difficult to navigate. There are now countless tools that allow people to experiment with marketing and product design. The scope of growth science has also greatly expanded across disciplines–the second pillar of growth science– from marketing to product implementation and operations. The application of this methodology has also expanded to include not only new customer acquisition but also to engagement and adoption. The optimization of the entire customer lifecycle is the third pillar of growth science.

We will cover the other two pillars of growth science in subsequent sections. Hopefully by explaining the three elements of growth science and providing a comprehensive guide, we can empower you to successfully employ growth science to supercharge your business.


Be sure to check back tomorrow for the next section: Growth Science Pillar 2: Optimization Across Disciplines! New sections of Growthzilla are published every weekday. 

Chapter 1: An Introduction to Growth Science

I remember looking at a picture with colorful tiles overlaid over a mock webpage. In the upper left there was a red-orange tile and another one right below. Across the top and down the left-hand-side of the page there were more orange and bright green tiles. As my eye gazed across to the right and down the mock page, the tiles got blue and finally purple. I was at Google in 2005 where my job was helping website owners figure out how to implement Google AdWords ads on their site. The picture that I was inspecting was a heatmap, which was derived from aggregate statistical data showing which ad placements resulted in the highest number of clicks.

I was astonished. I realized that design on the web was a big statistical game. Users had tendencies such as where they tended to look first on a screen and what colors tended to catch their eye. It was just a matter of using experimentation to figure out what those tendencies were and designing products that played to those preferences rather than worked against them.

This idea that we can test the effectiveness of product design resonated with me. Having studied physics and economics in college, I had a strong preference for proof and hard numbers over subjective intuition. My academic background led me to adopting a data and research-driven approach to designing user experiences for interactive products like websites.

I soon discovered testing tools such as eye-tracking, which allows one to track what a user is looking at on the screen, as well as A/B testing, which allows one to compare how effective two versions of a page are in relation to each other. I also read results of experiments that others were performing with headlines such as “Placing the security logo on the upper left between the search box and the navigation bar increased conversions 8.83%.” 

Later, during my tenure as director of user experience in a Silicon Valley startup in 2007, I decided to use experimentation to drive product design. The startup was a document sharing site that was one of the most visited web properties in the world. I suggested to Senior Director of Product that we should try experimenting with the registration button on the home page. He laughed and told me not to waste my time, but luckily the founders were supportive of the idea, and we did it anyways.

One of the senior engineers (a brilliant gentleman that later became a very successful entrepreneur) developed our in-house A/B testing platform. First we figured that maybe moving the registration button to a more prominent place might help our conversions–the proportion of people clicking through to the registration page. So we moved the button from all the way on the right of the menu bar to the center. Lo and behold, our conversion rate increased by more than 70%! I was stunned that simply moving the button to the left of the page by a few hundred pixels (effectively about two inches on a monitor) made such a huge difference. I think we all were.

Given our initial success, we decided to keep going to see what other optimizations we could make through testing variations. We next decided to change the design of the button itself. Originally the button was a bland shade of blue, so we decided to try making it bright green based on the hypothesis that users would more readily notice a brighter color and would, thus, be more likely to click it. Another round of A/B testing revealed that the green version gave us about another 50% increase in conversion rate. This moment in time changed my life. At that moment, I thought that soon everyone would be experimenting and testing product design because it had such unbelievable potential to help businesses grow.

Over the next decade, two important trends occurred. First, leaders figured out ways to experiment with changes to not only product implementation but also to marketing and operations such as customer service. Second, many new tools appeared in the market allowing businesses to experiment with things like marketing messaging and even customer service process. Some companies have taken full advantage of the methods and tools to dominate their competitors. Unfortunately, these exciting advances have not yet reached a mass audience, and many businesses continue to fall behind.

In speaking to friends, clients, and colleagues, we heard the same refrain: the possibilities seem so numerous and the methods so opaque that many folks simply do not know where to start. In addition, many people had heard of terms like “growth hacking,” and incorrectly believed that they should replicate hacks that other businesses have used rather than adopting a broader process that they can contextualize to their business and implement to reliably grow their customer base and revenue. Our aim in writing this book is to give everyone an overview of the three pillars that make up growth science as well as practical tactics for growing your business.


Check in tomorrow for the next section of Chapter 1.