Chapter 4: Evaluation Methods in Growth Engineering

There is a fundamental tension between speed and accuracy in growth engineering as there is in many other pursuits. On one hand, improvements are usually multiplicative, so the rate of growth increases with the pace of implementation. On the other hand, if you make changes that squash growth, progress will creep to a halt, so it’s important that you are quite certain those changes that you decide to keep are constructive. In this chapter, we will examine a number of different evaluation methods and discuss their merits as well as drawbacks, so you can develop a good intuition for when to employ those methods that will increase the likelihood of success within your resource constraints.

I like to think of the process of experimentation and testing as similar to finding one’s direction using a compass. There exists an optimal route from point A to point B, but without any guidance, the world of opportunities is three hundred and sixty degrees wide; the highest growth path could be anywhere. The challenge is that the needle on this compass does not tell you where to go with perfect certainty. Different evaluation methods provide you with various kinds of information and different levels of certainty about which course is best. Every time you carry out a new experiment and evaluate the outcomes, you get a new reading on which steps to take next or even when to step backwards.


Pie Wedges Confidence Level

The entire process of growth engineering starts with identifying what specific things about marketing, product implementation, or operations are not performing optimally and could be improved. Once you have identified what might be improved, the next step is understanding how they might be improved. The last step involves implementing enhancements and evaluating whether those changes actually caused positive changes to your growth metrics. Fortunately, there are methods that allow you to answer all three questions: what can be improved, how it can be improved, and whether it was improved. Evaluation methods in growth engineering can be sorted into three main groups: descriptive (what), qualitative (why), and experimental (whether).

Types of Research - Qualitative vs Quantitative

Descriptive analysis is a form of quantitative research and tend to tell you what could be improved. An example of this is examining your website’s analytics reporting and finding that a product page has a very high bounce rate, meaning that many people leave right after landing on this page. This simple analysis implies that the product page (the what) is potentially a problem area. Simple analysis helps you identify a specific issue, but it does not provide more impactful insight, such as showing that a certain change caused a given outcome.

Qualitative evaluation is a group of activities that center around understanding why certain things might be suboptimal as well as how they can be improved. These methods are based on a conversation with the customer through direct interviews, observation, and surveys. Whereas descriptive analysis is a kind of smoke test, qualitative methods are more akin to investigative journalism. In the example above, you might interview website users to understand how they perceive the product page on your website. What do they find compelling, confusing, or missing? This kind of information will help you gain insight for how your product or customer experience can be fixed by understanding why it falls short.

Having employed descriptive evaluation to spot areas that could be improved and discovering likely ways to optimize them through qualitative methods, you are ready to implement specific enhancements. Continuing with the above example, let’s say that many customers that you interviewed stated that the imagery on the product page was not very compelling and didn’t really give them a sense of the product that you’re selling. Armed with that information, your team adds more compelling images of the product. Going back to your website analytics, you find that more people do seem to be staying on the page. You’ve succeeded, right? Not necessarily. It just so happens that you made this change in the weeks leading up to Christmas, so are customers more engaged because of the change that you made or because they are much more motivated to buy a present for their loved ones?


Different Evaluation Methods

In order to evaluate if the changes that you made caused corresponding improvements in your growth metrics, you will likely need to experimental or quasi-experimental methods. The most common of these is A/B analysis given that many contemporary tools allow one to test two versions of a web page or marketing email by randomly displaying each version to a customer and measuring key metrics. Unfortunately, despite its popularity, A/B testing does not lend itself to certain contexts such as physical spaces, products, and human interactions. In those cases, businesses might have to turn to field experiments such as randomized control trials. The challenge is that those kinds of evaluations can be extremely costly and consequently should only be employed when the highest level of certainty is required.

The great balancing act here is between cost, time, and certainty. Let’s say that you are heading the growth team at a national bank and you find, through a survey, that many customers are unsatisfied with their experience at bank branches. Your team goes out to interview customers at a number of branches and find that certain themes emerge and customers tend to complain about the same things such as the long lines, lack of seating, unfriendly staff, and even the cold, industrial interior design. You decide that the first thing you’re going to try is training bank staff to act friendlier with customers. Now, how should you evaluate if this change positively affected the business. Do you need a large, fancy randomized control trial? Not necessarily. More positive sentiment in a survey or in customer interviews might be enough to convince you to implement this training nationwide. Then again, perhaps this training program will be extremely costly, and the executive team wants you to be dead-certain that it will be worth it. In that case, the only option might be a more rigorous statistical analysis.

The goal of the following sections is to educate you about common evaluation methods, their benefits, and their drawbacks.


This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Thursday to learn about descriptive evaluation methods. New sections of Growthzilla are published every week.

3.4 Measuring Non-Digital Marketing Channel Metrics

Some people seemingly default to easy-to-measure marketing channels such as pay-per-click advertisements rather than choosing other channels such as print and television advertisements, whose effectiveness can be more difficult to track. However, you should not let convenience drive you to waste money on less effective channels when, with a little effort, you can also measure, test, and refine more impactful forms of marketing. It’s worth spending some time to discuss a few tactics that will allow you to measure non-digital marketing campaigns.

The key thing to remember when testing different channels is to keep your marketing message the same, so the only thing that you’re varying is that channel itself. Otherwise, you won’t know if the difference in outcomes is due to one channel being more effective than the other, the message resonating more with the audience, or a combination of the two.

Ways to Measure the Effectiveness of Non-Digital Marketing Channels

3.4.1 Simple Survey

There are many opportunities to survey customers, including immediately when they visit your website, open your app, contact your sales team, or complete a purchase. The timing of the survey greatly depends on what you intend to measure. For example, one great way to ascertain what marketing channel most effectively drove new customers is simply to ask them in a survey at the time of purchase.

As mentioned in the first chapter, marketing not only determines the number of prospective customers a product attracts, but also the likelihood of purchase (acquisition) and retention. For example, we could launch an online marketing campaign that falsely exaggerates the capabilities of a product. This campaign could drive lots of prospective customers to our website. Depending on the effectiveness of the website, we will then convert a fraction of these people. Unfortunately, due to the nature of misinformation that brought these customers in, we immediately see large drop-off rates. This example demonstrates three critical stages: initial interest, purchase conversion, and post-purchase retention. We could implement surveys at all three stages to measure how well the marketing campaign drives prospective customers to your product or website, how effective it is at targeting the right customers and conveying accurate information leading to conversion as well as retention.

If the key question that you are trying to answer relates specifically to the effectiveness of driving potential customers to your product then you will want to survey them at the very beginning of the sales cycle when they come to your website, when they contact your sales team, and so on. On the other hand, it is sometimes much more interesting to consider both how well different marketing methods convert interest to actual usage or sales. In this case, you need to survey people after they have used your product, registered for your service, or completed a sale. Finally, it is also useful to have a sense of how well different campaigns attract the right customers and retaining them. In that case, you might want to survey customers some time after they have bought or started using your product. It’s important to acknowledge that customers may not recall the marketing campaign that drove them to try the product, and the data that you get from surveying individuals a long time after they have been exposed to the campaign is likely to be less precise.

Imagine that you are interested in learning how many people that bought your product were driven by your radio advertisement versus an online advertisement. Let’s assume that you are spending $10,000 on online advertising and an equal amount on the radio campaign. After surveying customers post-purchase you find that roughly twice as many people bought your product after hearing the radio advertisement versus those that saw the online advertisement. You compare the two channels based on your survey data and estimate the cost per acquisition as in the table below.

Example Comparison of Radio vs Online Advertising

It seems that the more traditional medium, radio, is a much more effective channel for attracting and converting customers, but how confident should you be in this analysis. You were very careful to keep the messaging the same for both channels, so that should not have affected the difference. But you are concerned that maybe there is a sampling bias where those customers that bought your product after hearing a radio advertisement are more likely to take the survey. This is a legitimate worry, and might have been hugely important if the numbers were closer. However, it is not necessary to be perfectly scientific about this kind of analysis when running growth experiments. It is certainly better to do a more thorough statistical analysis and take greater pains to design a more accurate survey, but such a level of rigor might not be worthwhile in many cases.

3.4.2 Special Landing Pages, Emails, or Phone Numbers

Another very clever way to measure the effectiveness of marketing channels is to tie them to unique website pages, telephone numbers, or even email addresses. For example, you might run a radio advertisement for your product, Sergio’s Widget, and direct folks to navigate to “” You might also run a television advertisement which directs people to “” The great thing is that you can be quite certain that folks that navigated to the landing page associated with heard your radio advertisement and not your television advertisement.

This method has some advantages over surveys. First, you do not have to interrupt customers while they are in the process of evaluating or buying your product. Instead, you send them down discrete paths from the very beginning. Second, people might misreport in a survey what piece of marketing led them to discover your product, whereas providing them with a unique link or phone number for each marketing channel forces them to remember more accurately. Third, and most importantly, providing them with a special destination opens up many opportunities to track them throughout the entire acquisition process. In other words, you can track how many people that started at went on to successfully complete a purchase. This is hugely valuable information.

Tracking marketing channels with unique web addresses and phone numbers also has some disadvantages of which you should be aware. Just as was true with surveys, this measurement technique will give you relative measures of performance. In other words, you might learn which marketing channel is likely to perform best. It’s also very likely that users will navigate to even though the radio advertisement mentioned This might not be a big issue and will not severely skew your relative measures of channel performance if we can assume that this problem occurs uniformly across the various channels. However, if one destination is much easier to remember than the other, you’ll likely have an inaccurate outcome. For example, if the telephone number that you provide in the radio advertisement is 1-800-888-888 while the television advertisement mentions 1-800-289-0973, it will seem that the radio channel is a lot more effective, relatively speaking, than it really is because the phone number is a lot easier to remember. What this all means is that you need to carefully select the unique identifiers that you use so that they don’t end up skewing your results.

3.4.3 Promotional Codes

Another great method for keeping track of acquisition across various marketing channels is providing promotional codes that are specific to each channel. This method is very similar to providing distinct web addresses, phone numbers, or emails, but it is different in the sense that people do not have to go to unique destinations. This can be a big advantage for prospective customers since they don’t have to remember a special web address or phone number. The main disadvantage is that the promotional code must be tied to some kind of discount or reward, otherwise people do not have any incentive to enter it, and this can be costly.


This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Thursday to learn about evaluation methods that you can use for growth. New sections of Growthzilla are published every week.

3.3 Retention Metrics for Growth

Example Average Lifetime Calculation

Imagine that you are really great at attracting new customers. However, the vast majority of the folks that you convince to try your product use it only a small handful of times and drop off. You don’t want to be losing customers as soon as you get them since your acquisition efforts will be squandered. This means that you have to focus on improving how well your product and business retains customers. But how do you know what tactics improve retention? You will need some way of measuring customer retention, so you can discern what changes are helping your business to keep existing customers. Below are three common measures that are useful to track.

3.3.1 Customer Retention Rate

The most common retention metric that businesses track is the customer retention rate (CRR). The customer retention rate is the percentage of customers that you kept during a period of time as compared to all the ones that you had at the beginning of the same period, excluding new customers that you gained. Mathematically, this percentage can be represented as follows:

The closer the number is to 1, the better your product and business are at retaining customers.

3.3.2 Average Customer Lifetime

Directly related to customer retention rate is the average customer lifetime. This is a fairly simple measure that captures how long your customers stay active, on average. To calculate this figure, one would take all of the customers that became inactive and average the period between when they first became customers and when they ceased to be customers. A simple example is shown in the table below.

Example Average Lifetime Calculation

There are a couple of things to note from the above example. First, it’s clear that this is a contrived example because we’d probably have many more customers that joined and fell off in the implied time frame. In your analysis, you would want to include all those customers that you acquired up to the present date. Second, those customers that are presently active should not be included in the calculation since they might skew the average if your acquisition rate is accelerating. Third, it can be difficult to identify which customers are active and which are not. For example, if your product is a real estate app just because a user has not logged in for a while might not mean that they have abandoned it.On the other hand, if your product has a monthly subscription fee, your customers will cancel it when they no longer want to use your product.. It is important to carefully pick the criteria to use for designating a customer as inactive as well as to consistently use those criteria. The final point is that it’s possible to compare the average lifetimes of cohorts of customers. In fact, that is the main way that your team could determine if the changes that they make increase retention by a longer average lifetime.

3.3.3 Dollar Retention Rate

Another very useful measure of retention (and perhaps a more accurate one) is the dollar retention rate, or DRR for short. Here is why many consider it to be a more accurate measure of the health of your business: imagine that you have a hundred customers and each of them pay you a hundred dollars a month for your amazing project management software. Your revenue is $10,000 per month. Now let’s say that you raised your monthly price to two hundred dollars per month. However, many of your current customers got upset and quit using your product. Specifically, thirty customers stopped using your project management app. Also, during that same month you got ten new customers paying the higher fee. Was raising the monthly fee a sound business decision?

The customer retention rate would have clearly fallen, but perhaps your company is making more money. The dollar retention rate is precisely the measure that could help answer the above question. Mathematically, DRR can be represented as:

Dollar Retention Rate DRR Calculation

Your customer retention rate given that thirty customers left and ten were added comes out to 0.7.

Customer Retention Rate CRR Example

At the beginning of the month you were making $10,000 and at the end of the month you are making $16,000 (70 X $200 + 10 X $200) with $2,000 in new revenue, which gives you a DRR of 1.4.

DRR Example

At the end of the day, your business is about revenue not about the total number of customers, so DRR might be the more authoritative gauge of retention. However, that does not mean that you should neglect your CRR. If you keep alienating customers by raising prices, you are probably not creating a viable long-term business model.


This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Thursday to learn how to measure performance of non-digitial marketing channels. New sections of Growthzilla are published every week.

3.2 Engagement Metrics for Growth

Unless your business is selling very high-ticket items such as real estate or luxury yachts, which most people buy rather infrequently, you probably want your customers to use your product consistently and intensively. If your customers use your product all the time, it’s a sign that it delivers a lot of value to them, and they will probably keep paying for it or using it. On the other hand, if customers hardly use the product, that signals that your business is providing very little value to them. Perhaps the product is not a great solution for their needs, they are finding it difficult to use your product, or the support that your business provides is very suboptimal and frustrating. In any case, those are serious problems that will probably sink your business in the long-run.

3.2.1 Ratio of People Performing Key Actions

Products exist and thrive in the marketplace because they solve challenges better than other solutions allowing customers to perform actions that are valuable to them. However, such actions vary for each product. For example, a social media app such as Instagram depends on people sharing pictures, reacting to them, and interacting with other. Those responsible for growth at Instagram might be tracking actions such as the number of new posts, the number of replies to posts, the number of likes, the number of times a user shares posts, or even the number of friend invites that an average user sends. On the other hand, if your product is a SaaS project management software, you might want to track actions such as the average number of times that tickets are assigned to team members, the number of project tasks created and closed, or the number of times that team members comment on a task.

3.2.2 Daily, Weekly, Monthly Active Users

Another very useful way to measure engagement of a product is how many people use it in a given span of time. Let’s imagine again that your company has build an online project management tool. What if a 10,000 users created an account, but only a hundred users log in during a given month? The absolute number of registered customers or even customers growth tall you nothing, but measuring how many of those folks use your product in a time period to the total customer base can tell you a lot about how engaging your product is.

The three common measures are daily active users (DAU), weekly active users (WAU) as well as monthly active users (MAU). These metrics capture the total number of users that performed some action, such as signing in or creating a post, that would deem them to be “active.” For example, if you define active users as those that log into your app, and you have 1,000 daily active users, that means that 1,000 users logged into your app during that given day.

The main challenge in this approach is determining what constitutes an “active” user. For example, is an active user one that comes to your app but does not do anything? Or is an active user that comes to your site (or opens your app), logs in, and does something such as creates a task or post? Your numbers will vary greatly depending on how you define “active,” but I would encourage you to be honest with yourself and define active users in a strict enough way that it will gauge engagement.

3.2.3 Ratio of Daily Active Users to Monthly Active Users (Stickiness)

A related metric is the ratio of daily active users over monthly active users, which measures “stickiness” of the product. Let’s consider the extreme cases of this ration to better understand how it signals the engagement of your product. If your product has 1,000 users that are active on any given day as well as 1,000 users that are active in any given month, that implies that the same 1,000 users were active each day for a whole month. In other words, they used your product every day. That is an enviable claim that even the most successful products in the world cannot make.

Consider now a case where you have about 33 daily active users and 1,000 monthly active users. (It’s worth noting that you can’t have less than 33 DAU because they would not add up to 1,000 over thirty days.) Your ratio of daily active users to monthly active users would imply that every day a new set of 33 users were active meaning that none of them used your product on more than one day in a whole month. It would be safe to say that you product is not very engaging.


This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Tuesday to learn about common retention metrics. New sections of Growthzilla are published every week.

3.1 Acquisition Metrics for Growth

As mentioned previously, the best metrics are ones that fit your business model, objectives, and success criteria. Nonetheless, below are a few acquisition metrics that are commonly used by businesses and can serve as a solid starting point for your own growth efforts. Each of the following metrics can provide a unique perspective on the levers that you can pull to stoke user growth, and it is worthwhile to try incorporating each of them in your process to see which strategic angle might yield the greatest gains. It is also important to note that your growth team can customize and build upon these standard metrics to really capture your unique business and growth models.

3.1.1 Conversion Rate

Probably the most common metric that growth leaders track for online products and apps is conversion rate. In a generalized form, a conversion rate measures the proportion of people that completed a process after starting it. It is important to really pay attention to what events are considered as the “start” and “end” of a process. For example, if you were measuring the conversion rate for registrations, what two events are you actually comparing? Are you comparing the number of people that successfully submitted a registration form to those all those that came to your website or just to those that came to the registration form? Or maybe then successful action is not just submitting the registration form but rather also being approved. Some common conversion rates are the proportion of people that:

  • Saw an online advertisement and navigated to a website,
  • Saw a call to action and navigated to a high-value page or screen,
  • Started the registration process and successfully finished it, and
  • Added an item to their cart and successfully purchased that item.

Some conversion rates can measure the effectiveness of your acquisition efforts as well as engagement, so it is important to consider what actions you are tracking and how they relate to business functions such as marketing, product development, and operations. For example, the conversion rate of people that started a registration process and successfully finished it might depend mostly on how the registration process is implemented rather than on the marketing that drove people to the registration page.

Conversion Rate Formula

At the same time, it’s very likely that the conversion rate for the registration process is also a function of the effectiveness of the marketing efforts. Let’s imagine two scenarios for an online project management software. In one scenario, marketing does a great job communicating the benefits of the tool to prospective customers. In another scenario, marketing does not communicate the value proposition very well. It’s reasonable to assume that those individuals that were exposed to the more effective marketing campaign would be more likely to complete the registration process because they have a greater conviction that the project management tool will be valuable in their work.

Clearly, the conversion rate for the registration process is probably a function of both product implementation (how easy it is to register) and marketing effectiveness (how convincing is the marketing messaging). Other conversion rates such as the proportion of people that see and click on online advertisement can capture the effectiveness of your marketing message and channel alone. The important point to remember is that if the conversion rate you pick could be affected by more than one function (marketing, product implementation, operations), your team should carry out your experiments on one area at a time to isolate and optimize their effects. For example, if you believe that the registration rate is affected by both how convincing the marketing message is as well as how easy it is to actually register, you would likely want to try optimizing marketing and product implementation separately rather than at once.

One question that business leaders typically have is what is a “good” conversion rate? It’s impossible to provide a general benchmark because conversion rates vary not only by product type but also by what you are trying to measure. For example, the conversion rates for seeing and clicking on online advertisements tend to be in the single digits while conversion rates for registration are likely much higher. In fact, it is really not useful to compare your conversion rates to benchmarks. Rather, realize that they can always be improved, and aim to constantly be ratcheting them up.

Another important point to understand is that conversion rates in themselves do not paint a holistic picture of growth performance nor do they provide insight to all the opportunities that exist. It is important to look at other metrics such as the customer acquisition cost and the rate of visits to key pages to gain a broader perspective on optimization possibilities.

3.1.2 Customer Acquisition Cost (CAC)

Another common metric used to gauge the effectiveness of a company’s acquisition efforts is customer acquisition cost (CAC), which is primarily an indicator of the efficacy of the company’s marketing efforts. As an example, imagine that you are using two channels to advertise your product, pay-per-click as well as online video advertisements. You do the numbers and find that the video advertisements have a conversion rate of five percent while the pay-per-click advertisements have a conversion rate of only two percent. If we based our growth engineering solely on conversion rate, this would be an open-and-closed case. Clearly, video wins, right?

It’s not necessarily true that video is the better marketing channel. Simply put, it could be that video costs way too much even though the conversion rate is much better. The video production costs and the ad placement costs can add up to the point that what you are paying for an actual sale is inferior to what you would have paid had you advertised with a pay-per-click ad. Even worse, it could be that you are losing money on every new customer because your cost per customer acquisition is less than the average revenue for that customer.

Customer Acquisition Costs Scenarios

Let’s consider an example, since it may not be intuitive how marketing with a higher conversion rate can be less worthwhile than ones with a lover conversion rate. Imagine that you are marketing a project management software over two channels: online video and pay-per-click (see table above). Which one is more effective?

For your video campaign, you had to hire an advertising agency to make the video, which cost you a total $50,000. Over the course of your the campaign, the video advertisement was displayed 100,000 times, and it cost you one cent every time it was displayed (cost per impression) for a total of $1,000. Of the 100,000 times that your video was displayed, people clicked on the link in the advertisement 5,000 times, which gives you a click conversion rate of 5%. Taking into account the production costs ($50,000) and the placement costs ($1,000), you have spent $51,000 to get 5,000 people to click on the advertisement and navigate to your site. Your cost per conversion is $5.10 assuming that we measure conversions as the proportion of people that click the link in the video and navigate to your site.

Let’s now consider your pay-per-click campaign. You have experience running pay-per-click advertisements, so you create the campaign on your own without having to hire expensive outside help, which makes your production costs zero. Just like the video campaign, your advertisement appears 100,000 times and gets 2,000 clicks for a 2% click conversion rate. In addition, it costs you $1.00 every time someone clicks the link in your advertisement and navigates to your site, which adds up to $2,000. Your net cost per conversion is simply the cost per click or $1.00. That means that even though the conversion rate on the video ads was twice as high as that for the cost-per-click campaign (5% vs 2%), the cost per conversion on the cost-per-click is less than one-fourth of video ($1.00 vs $5.10).

Should you invest your money in the pay-per-click campaign? Remember that customer acquisition cost is measure how much marketing spend it takes to get a converted customer. Clicking on an advertisement does not mean that individual will become a registered or paying customer. Let’s factor this into our fictitious example.

Let’s say that you are tracking customers that come from your video vs. pay-per-click campaigns. You find that for every two people that came from the video advertisement, one actually buys a subscription to your software whereas only one in four converts to an actual sale if they come via the pay-per-click advertising. Taking into account the conversion rate between a click and a purchase, you get a customer acquisition cost from video to be $10.20 versus $4.00 for the pay-per-click campaign. The campaign with the lower click conversion rate is actually more efficient! In fact, if the net revenue per customer is $10.00, you would be losing money on the video campaign. That is why conversion rate is extremely useful, but cost per acquisition is also critical in deciding which messaging and channel are optimal. Having established the conversion rates and customer acquisition costs for your marketing campaigns, you might want to set growth goals for your customer base. A great metric to track that goal is the rate of new customer acquisition, which is covered next.

3.1.3 Rate of New Customer Acquisition

You have two thousand paying customers. Is your company doing well in acquiring new customers? It’s difficult to tell. Perhaps your business had 1,900 customers a year ago and the customer base has barely grown. The raw number of customers does not necessarily provide a ton of insights. A much more useful measure is the rate of new customer acquisition, which tells you how well your acquisition efforts are performing in a given time period.

Customer Acquisition Rate Formula

By comparing customer acquisition rates you can tell if your business is acquiring more users now versus another historical time period. Everyone wants “hockey stick growth.” That is simply an accelerating growth rate, which means that you want your customer acquisition rate to be increasing with time. What is even more brilliant is that the customer acquisition rate can be combined with the churn rate, which is a measure of how many customers are leaving in a given time period, to provide a net customer growth rate. In other words, you want your customer acquisition rate to be higher than your churn rate, which is a measure of the effectiveness of your engagement and retention efforts.

The customer acquisition rate is a powerful top-line metric, since it measures what you ultimately want to increase: customer growth. While it is important to also track metrics that give you deeper or more specific insights into parts of your growth efforts, your team should always be tracking customer acquisition rate since not matter what the other numbers say, they better be collectively pushing the customer acquisition rate up.

3.1.4 Rate of Visits to Key Pages or Screens

Another common metric to track for online products and mobile apps is the rate of visits to key pages or screens as compared to overall visits to the site or user sessions on an app. Some businesses make the mistake of tracking just top-line metrics such as the customer acquisition rate explained above. This is a problem because actual purchases are preceded by many interactions and steps that contribute in aggregate to the overall acquisition or purchase rate. Understanding how customers progress through each step in the overall process allows your team to optimize the constituent parts of the full flow. For example, your team might be interested in the rate of visits to the pricing page to see if improvements in the navigation has resulted in more potential customers getting to the point where they evaluate the cost of the product or service.

By combining broad metrics such as customer acquisition costs and customer acquisition rate with specific and targeted metrics such as various conversion rates and visit rates to key pages and screens, your team will be equipped with data that can inform small changes that drive your progress toward your overall charted course.

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Tuesday to learn about common engagement metrics. New sections of Growthzilla are published every week.

Chapter 3: Tracking the Right Growth Metrics

3.0 Good Growth Metrics

Without good data, you won’t be able to reliably analyze the results of your growth experiments, and you’ll be back to a guessing game. Assuming that your data collection methods are sound, you still have to ensure that you are capturing the right metrics — ones that can help you understand whether an experiment has met its objective as well as your success criteria. In this chapter, we will review the characteristics of good growth metrics, and some standard metrics that can get your team started.

There are many qualities that make metrics “good,” but key characteristics to look for are the following:

  • Captures your objective
  • Easy to measure
  • Accurate
  • Actionable
  • Predictive
  • Based on your growth model

Captures Your Objective

Some might claim that it’s intuitive which metrics are appropriate for certain experiments, but this is not true. Experiments are meant to meet certain objectives and satisfy success criteria that you set. For example, there is often a tension in trying to optimize customer acquisition. Should you focus on quantity or quality? You can cast a bigger net to get more customers, but those customer probably won’t be as engaged as customers that you could get through more targeted acquisition efforts.

If your objective is just to grow the raw number of customers, you might track metrics such as the rate of new registrations or the conversion rate, which captures the proportion of people that finish the registration process to the number of people that start it. However, if your objective is to build a customer base of very engaged customers, you might prefer to measure things such as your daily active users, which captures how many of your users spend time with your product each day. Another metric that might be appropriate for capturing your objective to build an engaged customer base might be the average time that a customer is using your product in a given week. As you can see, the metrics that are appropriate for each objective are as disparate as the objectives themselves, so it’s important to take the objective into account when picking a metric to capture.

Easy to Measure

With the digital age and the proliferation of tools, many metrics have become easy to capture. However, some metrics remain difficult to capture. Those data points tend to relate to the non-digital realm such as measuring the acquisition rate of customers that saw a billboard advertisement or the retention rate of people that saw a television advertisement. Often times, it’s possible to measure the effects of a given experiment using a number of different metrics, and it’s worthwhile to focus on those metrics that are easier to measure to ensure that you do not waste effort by running experiments without good data outcomes. However, it’s important to understand that there is a danger of defaulting to those metrics that are easiest to garner and not those that provide the best basis for analysis. You will need to determine the right balance for your goals. Sometimes it is preferable to opt for difficult to measure data if it will give you a clearer picture of whether or not an experiment successfully met your objective.


Good metrics should also be strongly tied to outcomes rather than loosely represent them. For example, let’s say that you wish to impact how many purchases customers make. You could choose to measure either the customer’s intent-to-purchase or actual purchases. It is well known that intent to purchase is not always an accurate indicator of actual purchases. Therefore, you would likely be better off measuring the actual volume of purchases rather than a virtual metric such as intent.


All metrics should be actionable in theory, but some are definitely easier to act upon than others. How actionable metrics are greatly depends on how specific they are. Let’s say that your product is an online, subscription-based task management tool. If I told you that the average customer satisfaction score for your product is 6.9 out of 10, would you know immediately what to do next? What if I told you that 4 out of five 5 customers that register for your product start but never finish creating their first task entry? Would you have a better idea of where to focus and what to try next? I hope that the second metric would arm you with much more actionable data. It takes a lot of time and resources to conceive and implement an experiment, so it’s well worth it to give some thought to how actionable the data that you intend to measure will be.


It is always a good idea to track data that gives you insight into your business and are outside the context of a particular experiment. These metrics are different in nature than those that you should be tracking to identify if a particular growth experiment has met your objective or success criteria. Instead, the data should help to shape your growth strategy by helping you anticipate problems before they become hugely disruptive to your business. A classic example of this is the distinction between the rate of customer service emails and customer churn rate, which indicates how quickly existing customers are abandoning your product.

Let’s go back to the example of an online task management software. Imagine that your team changes a big piece of your product, such as, how tasks are created by users. Should you be tracking the customer churn rate or the rate at which your customers are contacting customer support. The former is usually calculated after the fact, whereas you can see a spike in customer support inquiries immediately. Therefore, the rate at which your customers are contacting customer support will let you react much more quickly than the churn rate, so you should be keeping your eyes on customer support emails or calls. As you think about what metrics you should track to give you general insight into the health of your business try to think about what data can tip you off about problems earliest.

Based on Your Growth Model

Although metrics such as daily active users or conversion rates are fine, nothing beats creating a data capture strategy based on your own growth model. Doing so ensures that you have precisely those metrics that are tied to variables that most directly affect the growth rate of your business. Your revenue and customer growth models should tell you precisely what metrics your team should be tracking.

Marketplace Revenue Growth Model

Let’s consider an online marketplace with the above model as an example. At first glance, you can see that you should be capturing the following metrics:

  • The number of sellers
  • The average number of posts per seller per time period
  • The number of views of items for sale
  • The number of buyers
  • The number of times an average buyer visits the marketplace in a given time period
  • The ratio of the number of times that buyers visit an item page to the number of completed sales
  • The average fee that your company charges per transaction
  • The average sale price of items in the marketplace

Not only did the revenue model help us identify novel metrics that are perfectly suited for an online marketplace, such as the number of views of items for sale, but it also helped us make standard metrics, such as conversion rate, specific and contextual to the business.

As mentioned above, making changes to marketing, product implementation, and operations is resource intensive, so it’s well worth spending some time to identify metrics that will help your team clearly determine whether your growth experiments meet their objective and satisfy success criteria to ensure that you are not wasting time and resources. Of course, it would be ideal if all of your metrics satisfied all of the criteria above, but the reality is that many will not. In that case, it’s best to focus on ensuring that the metrics that your team chooses to capture align with your model and are accurate. Beyond your tailored metrics, there are others that have become standard practice to track. We will explore these in greater details in the following sections.

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back on tomorrow to learn about common acquisition metrics. New sections of Growthzilla are published every weekday.


2.5 Iterating on Previous Growth Experiments

Iterative Experimentation is Key to Growth

This post is a part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

A key part of the growth science methodology is iteration. It’s not enough to form one hypothesis, test it, and implement a change. If you stop short, you’ll be missing out on substantial optimizations that can add up to incredible gains to customer or revenue growth. The key is to keep experimenting in areas where previous changes have led to strategically significant gains.

You can either iterate on past experiments and try small variations or try the same kinds of experiments in other contexts. For example, let’s say that you try an experiment to get more people to click on the registration button on your website. In trying to increase the conversion rate on new customer registration on your web app, you don’t want to stop with just changing the color of the “Register” button to bright green. You also should double down on this line of thinking and test changing the button color to orange and red. Not only that, you might try very related experiments such as testing the label text. Perhaps you can try “Sign up” and “Get Started” instead of “Register.” And these iterations are only for one button. Not only that we haven’t even started talking about marketing and operations.

A great example of iterative experimentation leading to markedly higher increases in growth comes from a leading language learning platform, Duolingo. Gina Gotthilf, VP of Growth, recounted the story of how her team made huge strides in Duolingo’s customer engagement through iterative experimentation with their registration process.

Read more

2.4 Running Successful Growth Experiments

Sample Growth Objectives and Experiments

This post is a part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

By utilizing experimentation to help us decide if changes to product, marketing, and operations are effective, we avoid having to rely solely on our intuition. Without experimentation, we would implement changes and hope that we’re right slightly more than half of the time. Rather than engineering growth, we would be relying on an art form, which would be dominated by a select few that had outstanding intuition such as Steve Jobs (or those who claim to have this level of intuition). Growth engineering would be inaccessible to the majority of us.

Experiments make growth accessible to nearly everyone because they follow systematic ways of testing hypotheses to reach specific outcomes and are not unlike experiments in the physical and social sciences. Admittedly, growth experiments are usually not as rigorous as in academia, but the fundamentals are still the same. Anyone that learns the basic experimentation methodology can lead successful growth development at their company. Of course, you will likely be more effective with greater experience and practice, but it’s important to learn strong fundamentals from the beginning.

In this section, we will review the key components of successful experimentation. The first step is understanding what are you trying to achieve with your experiment. Are you trying to improve how long users spend on your site or how quickly the can get their tasks done? Then we will consider how to measure whether or not the changes that you implement have successfully accomplished that objective, or not. Then we will discuss ways to create a sounds hypothesis about ways to reach your objective. Finally, we’ll review the nuts and bolts steps in running good experiments as well as gathering and analyzing results.

Read more

2.3 Strategizing and Prioritizing Experiments

Growth Strategy Cycle

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

When you model your business, you will likely find that there are many ways that you can potentially improve growth. On one hand, this is great news because you have many opportunities to grow your business. On the other hand, this is very challenging because you have two factors working against you: limited resources and a finite market size. Moreover, every change that you try will not work. This is why following a more scientific approach that includes forming hypotheses and measuring results is fundamental to growth science. Each experiment requires capital and human resources investment, and creating a well thought-out strategy and continually prioritizing experiments will be pivotal to your growth development efforts.


2.3.1 Brainstorm Growth Optimization Opportunities

As you begin to engineer your company’s growth, you will iteratively brainstorm new optimizations to try, create a strategy to guide your efforts, and constantly prioritize your growth experiments based on that strategy. Luckily, you have already created a framework that will help you to brainstorm and evaluate experiments. The customer journey map and growth model that you created will help guide your brainstorming. The journey map will highlight key actions in the customer interactions with your product and company, and can be juxtaposed with your growth model to understand how these actions affect overall customer and revenue growth.

Growth Strategy Cycle

Read more

2.2 The Customer Lifecycle

Events Customer Lifecycle

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

Let’s review the customer lifecycle before we dive too deeply into strategy since it will serve as the basis for a lot of the concepts in this book. A typical customer lifecycle consists of five parts: awareness, acquisition, engagement, activation, and retention. The journey through the conversion funnel is not always linear and could jump stages. For example, an individual might use a product (acquisition), get tired of the product and contemplate abandoning it (retention), and fall back in love with the product (engagement). Notwithstanding what journey an individual might take, we need to have a game plan for each stage.


Customer Lifecycle - Awareness, Acquisition, Engagement, Activation, Retention

2.2.1 Customer Awareness

It all starts with awareness. One study performed by McKinsey & Company indicated that it can take five to eight exposures to an advertisement for it to make people truly aware of a product. People can learn about your product in many different ways such as through traditional marketing, through friends and acquaintances, and through organic news stories. What might be less obvious to the reader is that the product itself can help spread awareness. For example, a social sharing feature can help current customers recommend your product to your friends. Your operations can affect awareness too. For example, your customers can learn about new products that your company is offering when they call customer support.


2.2.2 Customer Acquisition

After a person learns about your product, they might decide that they want to try or buy it. This begins the acquisition period. It might seem that acquisition should not be a phase but rather a short point in time, but that is rarely how people become customers. For mobile apps, a new user might have to sign up. That can be a long and complex process involving filling out a registration form, validating their email, and completing their profile. Potential customers can leave the acquisition cycle in any of those steps, and it is not until they get through the whole process before they become active customers.

For other products such as enterprise software, the acquisition process can be extremely drawn out and can involve a sales team that pitches the product, a technical implementation team that helps to answer technical questions and sets up the product, a training team, and so on. The acquisition phase starts with the intent to try or buy a product and ends when the customer actually buys the product or uses the product for the first time.


2.2.3 Customer Engagement

Just because a person has used or bought your product does not mean that they are a valuable customer. Even popular apps like Twitter have scores of users that post a few things and then never use it again. Those are not engaged users, and having unengaged users is bad for business. Usually customers fail to become engaged because they either can’t overcome the learning curve or the product fails to meet their expectations. Luckily, it’s possible to systematically and predictably fix both.

Some products have much more compressed engagement cycles. For example, if you are in the market for a luxury yacht, you probably won’t be making a bunch of repeat purchases. However, that does not mean that all yacht owners are equally engaged. Some owners love their luxury yachts, and it shows as they merrily sail around the Mediterranean and extol their yacht’s virtues to their friends. Those are the kinds of customers that you want!


2.2.4 Customer Activation

Activation is a big jump in the customer’s level of engagement. The anecdote that I like to tell is my relationship with Amazon. I became an Amazon customer in 2001, when I was a college student. I realized that buying textbooks on Amazon could save me a ton of money, so every term I bought a bunch of books on Amazon. I was a pretty engaged customer. I liked that I could save money, which was tight during my college years, and I was generally happy with the service.

Fast forward a few more years, and Amazon’s offerings broadened. At that point, I was a young professional, and I would occasionally buy a DVD or a book. Once again, I was a solid repeat customer, but I would only buy things that were difficult to track down in a physical store and for which I could wait a week to be delivered. I still preferred to run down to my local BestBuy to buy electronics, DVDs and videogames since I could get it right away.

Fast forward again to 2009 when I went back to graduate school, and Amazon gave me a free Amazon Prime membership since I was a student. That year I bought my textbooks on Amazon, but my purchase behavior started to change radically. Suddenly, I was buying everything on Amazon: DVDs, video games, electronics, clothes, household items, you name it. On reflection, my Amazon Prime subscription annihilated a huge deterrent: having to wait a week for my orders. I didn’t realize at the time what a big roadblock the shipping time was, but now that I had free two-day shipping the floodgates were open. In fact, there was a time when my sons were still babies, when I’d get Amazon shipments multiple times a week filled with diapers, baby clothes, household items. That is what I call Activation!

Activation is a phase in a customer’s lifetime when they become super engaged. Many customers never get to that stage, but that is where the biggest growth can happen.


2.2.5 Customer Retention

Unfortunately, all good things must come to an end. Even your most activated customers may get frustrated with your product or company and will choose to leave at some point. More likely, those customers that never got very engaged will abandon your product. The good news is that abandonment does not usually happen instantaneously. There are often signs that a customer is unhappy, and you will likely have ample opportunities to make your customers happy. Even better, you can take steps during customer acquisition and engagement that will prevent your customers from becoming unhappy with your product once they convert to active customers.

I was recently trying a new online project management app. The app and the company behind it did many things right to make sure that I would not get frustrated or disappointed as I became an active user. First, when I signed up for the app, they sent me a personalized email from an individual on their team thanking me for registering and letting me know that I can contact her at any point if I have any questions or issues. Second, when I logged onto the app for the first time, they had a great video tutorial giving me an overview of the features and best practices. On top of that, they had little contextual tutorial messages as I explored the app for the first time.

I set up a simple project to test out the software and invited one of my teammates. It was a pretty solid project management app, but there were a few little things that irked me, and I stopped using it with time. The company saw that I hadn’t used it in a while and they sent me a couple emails (automated, I’m sure, but signed by my customer service representative) asking me if there is anything that they can do to help me with the app. I finally relented and replied to the email and set up a call with my dedicated customer service representative that addressed each of my complaints in turn. She couldn’t solve all of them, but she did help me find other ways to get things done. The company not only did its best to avoid abandonment, but they also successfully retained me when I was not totally happy. I’m still a loyal customer.


2.2.6 Engagement Versus Retention

Not everyone will agree with defining retention as simply a measure of keeping customers. To some retention means the act of keeping customers coming back, while engagement is a measure of how intensely they use your product at any one time. For example, they would consider Google Search to be a product with low engagement and high retention given that users use Google Search briefly, but then keep coming back.

Consider a student that is writing a term paper, who uses Google to help them find resources and information, and then never uses Google again. Because they were doing a ton of queries while writing the term paper, we would say that their engagement was high but their retention was low because they never used Google again. I think the majority of people would not consider this person to be highly engaged even if they did submit dozens of searches in a day. Wikipedia users are similar. They use the online encyclopedia intensively but infrequently.

I find it more useful to think of engagement as a measure of how much a customer is using a product, which is a function of intensity and frequency, whereas retention indicates if I am still an active customer. For example, which user is most engaged, one that uses your product once every day or another that uses your product ten times a day every ten days? What about a customer that uses your product twenty times over the span of two days but never touches it again? I would say that the user that used it twenty times and never again is not engaged because he is not an active customer while the other two are roughly equally engaged.

It’s often very ambiguous whether to consider some customers as being active or having abandoned your product, and every company has it’s own cutoff point depending on the nature of your product. For example, many years ago, I signed up for Flickr to share my photos with friends and family but stopped using it after a while and have not logged on for over four years. I still have an account, so am I still an active Flickr customer? It depends on Flickr’s definition of what they consider to be an “active customer.”


2.2.7 How Acquisition, Engagement, Retention Work Together

In 2007, a recent Yale graduate, Justin Kan, came up with an idea to stream every minute of his life over the internet. With the help of Emmett Shear, Michael Seibel and Kyle Vogt, he created, which would become the world’s first prominent livecasting service. The team grew to be a fairly solid business employing about twenty-five people, but at some point the founders realized that their idea had hit a ceiling. There were only so many things that it made sense to livecast, and they had tapped out all of them. rose to about 30 million unique visitors per month, but the majority of their users weren’t super engaged, except for one tiny segment. Emmett Shear made the realization that video gamers, which made up just 3% of their user base, were fanatical users. They would sometimes spend hours streaming themselves playing video games and others would watch them playing for long stretches of time. The four founders decided to focus on addressing just the video game market and created a new version of called Twitch, which became one of the hottest startups in Silicon Valley and sold to Amazon for $970 million. had the perfect product all along, but they were distributing it to the wrong customer base. Not too many people were super engaged by watching Justin sleep or others sit around in front of the computer, but the video gamers were completely hooked. Once they focused on acquiring the video gamers their product growth exploded. This is one of the best examples of how acquisition, engagement, and retention are related.

Many people forget that the concept of product-market fit is composed of two parts and focus, instead, on just the product. However, it’s not just about building the perfect product. Finding the right customer whose needs your product meets most adeptly is just as important.

If you target the wrong customer, it does not matter how good your product is, you will always have problems engaging those customers because they won’t really have the problem that your product is trying to solve. It’s like selling goose down parkas in Miami Beach. You might have the most stylish, warmest goose down parka in the world, but if you sell it to folks in a place that is constantly hot, your customers won’t love your product and will eventually shove it in their closet or give it away to Goodwill. On the other hand, if you sold that same amazing down parka in St. Paul, Minnesota, your customers would wear it all winter long and would love it for years to come. Their engagement and retention would be off the charts.

To put it simply, if you are effectively targeting and acquiring the right customers for your product, they will appreciate it and will be highly engaged. Customers that are highly engaged, are very unlikely to abandon your product. How much do you like Google? I bet you’re a highly engaged Google user. Are you very likely to stop using Google. Probably not unless another even more amazing search engine comes along.


2.2.8 Key Points in the Customer Lifecycle

At the beginning and end of each phase of the conversion funnel are key events that demarcate the end of one and the beginning of another. These points are hugely important because you can influence these events to supercharge your growth.


Events Customer Lifecycle

The First Time Someone Hears About or Sees Your Product

A good first impression is trickier than you might think, and it will set the stage for the rest of your relationship with you customer. First, you want to make sure that you’re reaching the right people — those that have the problem that your product is solving. You also want to make sure to communicate how your product is better at meeting their needs or solving their problems. Finally, you should tell them how to get your product or what to do next. Without meeting these three fundamental awareness goals, acquisition becomes an uphill battle. Even engagement, activation and retention can be adversely affected by a poor first impression.

The First Time Someone Uses or Buys Your Product

Usability and customer onboarding make the biggest difference when it comes to creating a great first experience. There is not much that you can do if your product doesn’t have the right features to meet the needs of a customer. Certainly, you can add important missing features with time, but by that time many of the dissatisfied customers will be long gone. Not only that, as soon as you add a new feature, customers will want others, and you will be constantly playing catchup. However, I have found that for a significant set of the customers the right features do exist, but they are hard to find or understand how to use. Moreover, bad usability often forces customers to abandon products long before they learn their full utility. Both of these problems are preventable, and addressing them will go an extremely long way to improving your engagement and retention.

The Customer Uses Your Product More or Buys More

A user will very rarely discover how great your product is in their first use. It often takes many interactions to find out if and how well it will work for the customer. Using or buying a new product is an awful lot like moving to a new city. It takes a while to learn where the good restaurants, parks, and hangout spots are.

Often times we are pleasantly surprised about what the city has to offer. Other times we are disappointed to find that there is not much more to the new place than initially meets the eye. It’s not that the place has changed in the time since you moved; you just have learned it more deeply. Often people discover new things with the help of their friends or colleagues. With your product, you want to be that friend that guides their discovery. Your marketing, product, and operations should be aimed at helping the user discover and learn your product as quickly as possible, so they appreciate it for what it really is.

Customer Becomes a Habitual User or Buyer

The holy grail of growth is the point at which an individual becomes a habitual user or buyer. This is when they enter the activated phase of their customer lifetime. We all are very familiar with this concept. Earlier, I told my story about how Amazon was able to activate me, though free two-day shipping, to become a weekly shopper. If you are on Facebook, you are very likely a habitual user, checking your account at least a few times per week.

On the other hand, there are also many products that I use sporadically. For example, there is a personal finance software that I have been using for the past five years, but I only log onto it when my spending really gets out of hand. According to many reports, there are a great number of Twitter users that are like that. Folks sign up, write a few tweets, follow a few users and then hardly do anything else. On the other hand, there are many Twitter users that post seemingly every minute. The latter are the kinds of customers you want to cultivate with great product experiences, effective marketing, and world-class operations.

Customer Becomes Unhappy with Your Product

Even the best products in the world have unhappy customers. For example, the average customer satisfaction score in the United States is 87%.  That means that about 13% of customers are unhappy at any one time. If your product is a social media app, your customer satisfaction score is likely around 78%.

Disappointment and frustration can be factor of both product experience as well as marketing. You can have the best financial management tool in the world for middle income customers, but if you have high net worth or very poor customers, they will likely become unhappy with your product. On the other hand, you could be reaching the perfect audience, but perhaps your product is difficult to use. This will also not work in your favor.

There are also times when customers simply haven’t been able to figure out how to use your product to truly meet their needs. Luckily, that’s something that you can fix by systematically listening to your customers, experimenting with better ways to help them get the best of your product, measuring the impact that those changes have, and keeping only those that move the needle in the right direction.

Individual Stops Using the Product

Social scientists believe that the last impression is more important than the first. That is why you should make sure to end on a good note as much as you can. If you can do a good job saying goodbye to a customer, you might still get them back in the future. Perhaps they stopped using your product because it was missing a key feature, and when you finally build that feature you want them to try it again! Just as importantly, ending on a good note might mean that they will refer your product to their friends and colleagues. The might say, “It didn’t have this one thing that I really wanted, but it’s still a great product and a caring company!”


2.2.9 Marketing, Product, Operations through the Customer Lifecycle

Marketing, product design, and operations can each affect all stages of the customer lifecycle. It’s obvious that marketing can drive awareness, but we also saw with the above examples that product features and even customer support can drive awareness as well as acquisition. Conversely, how good a business is at turning interested folks into customers depends greatly on their ability to market to and draw the right customer. It is important to remember that marketing, product implementation, and operations could potentially affect all parts of the customer lifecycle as we consider how to engineer growth.


Marketing, Product, Operations over Customer Lifecycle

Be sure to check back tomorrow to learn about strategizing and prioritizing experiments. New sections of Growthzilla are published every weekday.