Chapter 3: Tracking the Right Growth Metrics

3.0 Good Growth Metrics

Without good data, you won’t be able to reliably analyze the results of your growth experiments, and you’ll be back to a guessing game. Assuming that your data collection methods are sound, you still have to ensure that you are capturing the right metrics — ones that can help you understand whether an experiment has met its objective as well as your success criteria. In this chapter, we will review the characteristics of good growth metrics, and some standard metrics that can get your team started.

There are many qualities that make metrics “good,” but key characteristics to look for are the following:

  • Captures your objective
  • Easy to measure
  • Accurate
  • Actionable
  • Predictive
  • Based on your growth model

Captures Your Objective

Some might claim that it’s intuitive which metrics are appropriate for certain experiments, but this is not true. Experiments are meant to meet certain objectives and satisfy success criteria that you set. For example, there is often a tension in trying to optimize customer acquisition. Should you focus on quantity or quality? You can cast a bigger net to get more customers, but those customer probably won’t be as engaged as customers that you could get through more targeted acquisition efforts.

If your objective is just to grow the raw number of customers, you might track metrics such as the rate of new registrations or the conversion rate, which captures the proportion of people that finish the registration process to the number of people that start it. However, if your objective is to build a customer base of very engaged customers, you might prefer to measure things such as your daily active users, which captures how many of your users spend time with your product each day. Another metric that might be appropriate for capturing your objective to build an engaged customer base might be the average time that a customer is using your product in a given week. As you can see, the metrics that are appropriate for each objective are as disparate as the objectives themselves, so it’s important to take the objective into account when picking a metric to capture.

Easy to Measure

With the digital age and the proliferation of tools, many metrics have become easy to capture. However, some metrics remain difficult to capture. Those data points tend to relate to the non-digital realm such as measuring the acquisition rate of customers that saw a billboard advertisement or the retention rate of people that saw a television advertisement. Often times, it’s possible to measure the effects of a given experiment using a number of different metrics, and it’s worthwhile to focus on those metrics that are easier to measure to ensure that you do not waste effort by running experiments without good data outcomes. However, it’s important to understand that there is a danger of defaulting to those metrics that are easiest to garner and not those that provide the best basis for analysis. You will need to determine the right balance for your goals. Sometimes it is preferable to opt for difficult to measure data if it will give you a clearer picture of whether or not an experiment successfully met your objective.

Accurate

Good metrics should also be strongly tied to outcomes rather than loosely represent them. For example, let’s say that you wish to impact how many purchases customers make. You could choose to measure either the customer’s intent-to-purchase or actual purchases. It is well known that intent to purchase is not always an accurate indicator of actual purchases. Therefore, you would likely be better off measuring the actual volume of purchases rather than a virtual metric such as intent.

Actionable

All metrics should be actionable in theory, but some are definitely easier to act upon than others. How actionable metrics are greatly depends on how specific they are. Let’s say that your product is an online, subscription-based task management tool. If I told you that the average customer satisfaction score for your product is 6.9 out of 10, would you know immediately what to do next? What if I told you that 4 out of five 5 customers that register for your product start but never finish creating their first task entry? Would you have a better idea of where to focus and what to try next? I hope that the second metric would arm you with much more actionable data. It takes a lot of time and resources to conceive and implement an experiment, so it’s well worth it to give some thought to how actionable the data that you intend to measure will be.

Predictive

It is always a good idea to track data that gives you insight into your business and are outside the context of a particular experiment. These metrics are different in nature than those that you should be tracking to identify if a particular growth experiment has met your objective or success criteria. Instead, the data should help to shape your growth strategy by helping you anticipate problems before they become hugely disruptive to your business. A classic example of this is the distinction between the rate of customer service emails and customer churn rate, which indicates how quickly existing customers are abandoning your product.

Let’s go back to the example of an online task management software. Imagine that your team changes a big piece of your product, such as, how tasks are created by users. Should you be tracking the customer churn rate or the rate at which your customers are contacting customer support. The former is usually calculated after the fact, whereas you can see a spike in customer support inquiries immediately. Therefore, the rate at which your customers are contacting customer support will let you react much more quickly than the churn rate, so you should be keeping your eyes on customer support emails or calls. As you think about what metrics you should track to give you general insight into the health of your business try to think about what data can tip you off about problems earliest.

Based on Your Growth Model

Although metrics such as daily active users or conversion rates are fine, nothing beats creating a data capture strategy based on your own growth model. Doing so ensures that you have precisely those metrics that are tied to variables that most directly affect the growth rate of your business. Your revenue and customer growth models should tell you precisely what metrics your team should be tracking.

Marketplace Revenue Growth Model

Let’s consider an online marketplace with the above model as an example. At first glance, you can see that you should be capturing the following metrics:

  • The number of sellers
  • The average number of posts per seller per time period
  • The number of views of items for sale
  • The number of buyers
  • The number of times an average buyer visits the marketplace in a given time period
  • The ratio of the number of times that buyers visit an item page to the number of completed sales
  • The average fee that your company charges per transaction
  • The average sale price of items in the marketplace

Not only did the revenue model help us identify novel metrics that are perfectly suited for an online marketplace, such as the number of views of items for sale, but it also helped us make standard metrics, such as conversion rate, specific and contextual to the business.

As mentioned above, making changes to marketing, product implementation, and operations is resource intensive, so it’s well worth spending some time to identify metrics that will help your team clearly determine whether your growth experiments meet their objective and satisfy success criteria to ensure that you are not wasting time and resources. Of course, it would be ideal if all of your metrics satisfied all of the criteria above, but the reality is that many will not. In that case, it’s best to focus on ensuring that the metrics that your team chooses to capture align with your model and are accurate. Beyond your tailored metrics, there are others that have become standard practice to track. We will explore these in greater details in the following sections.

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back on tomorrow to learn about common acquisition metrics. New sections of Growthzilla are published every weekday.

 

2.5 Iterating on Previous Growth Experiments

Iterative Experimentation is Key to Growth

This post is a part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

A key part of the growth science methodology is iteration. It’s not enough to form one hypothesis, test it, and implement a change. If you stop short, you’ll be missing out on substantial optimizations that can add up to incredible gains to customer or revenue growth. The key is to keep experimenting in areas where previous changes have led to strategically significant gains.

You can either iterate on past experiments and try small variations or try the same kinds of experiments in other contexts. For example, let’s say that you try an experiment to get more people to click on the registration button on your website. In trying to increase the conversion rate on new customer registration on your web app, you don’t want to stop with just changing the color of the “Register” button to bright green. You also should double down on this line of thinking and test changing the button color to orange and red. Not only that, you might try very related experiments such as testing the label text. Perhaps you can try “Sign up” and “Get Started” instead of “Register.” And these iterations are only for one button. Not only that we haven’t even started talking about marketing and operations.

A great example of iterative experimentation leading to markedly higher increases in growth comes from a leading language learning platform, Duolingo. Gina Gotthilf, VP of Growth, recounted the story of how her team made huge strides in Duolingo’s customer engagement through iterative experimentation with their registration process.

Read more

2.4 Running Successful Growth Experiments

Sample Growth Objectives and Experiments

This post is a part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

By utilizing experimentation to help us decide if changes to product, marketing, and operations are effective, we avoid having to rely solely on our intuition. Without experimentation, we would implement changes and hope that we’re right slightly more than half of the time. Rather than engineering growth, we would be relying on an art form, which would be dominated by a select few that had outstanding intuition such as Steve Jobs (or those who claim to have this level of intuition). Growth engineering would be inaccessible to the majority of us.

Experiments make growth accessible to nearly everyone because they follow systematic ways of testing hypotheses to reach specific outcomes and are not unlike experiments in the physical and social sciences. Admittedly, growth experiments are usually not as rigorous as in academia, but the fundamentals are still the same. Anyone that learns the basic experimentation methodology can lead successful growth development at their company. Of course, you will likely be more effective with greater experience and practice, but it’s important to learn strong fundamentals from the beginning.

In this section, we will review the key components of successful experimentation. The first step is understanding what are you trying to achieve with your experiment. Are you trying to improve how long users spend on your site or how quickly the can get their tasks done? Then we will consider how to measure whether or not the changes that you implement have successfully accomplished that objective, or not. Then we will discuss ways to create a sounds hypothesis about ways to reach your objective. Finally, we’ll review the nuts and bolts steps in running good experiments as well as gathering and analyzing results.

Read more

2.3 Strategizing and Prioritizing Experiments

Growth Strategy Cycle

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

When you model your business, you will likely find that there are many ways that you can potentially improve growth. On one hand, this is great news because you have many opportunities to grow your business. On the other hand, this is very challenging because you have two factors working against you: limited resources and a finite market size. Moreover, every change that you try will not work. This is why following a more scientific approach that includes forming hypotheses and measuring results is fundamental to growth science. Each experiment requires capital and human resources investment, and creating a well thought-out strategy and continually prioritizing experiments will be pivotal to your growth development efforts.

 

2.3.1 Brainstorm Growth Optimization Opportunities

As you begin to engineer your company’s growth, you will iteratively brainstorm new optimizations to try, create a strategy to guide your efforts, and constantly prioritize your growth experiments based on that strategy. Luckily, you have already created a framework that will help you to brainstorm and evaluate experiments. The customer journey map and growth model that you created will help guide your brainstorming. The journey map will highlight key actions in the customer interactions with your product and company, and can be juxtaposed with your growth model to understand how these actions affect overall customer and revenue growth.

Growth Strategy Cycle

Read more

2.2 The Customer Lifecycle

Events Customer Lifecycle

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

Let’s review the customer lifecycle before we dive too deeply into strategy since it will serve as the basis for a lot of the concepts in this book. A typical customer lifecycle consists of five parts: awareness, acquisition, engagement, activation, and retention. The journey through the conversion funnel is not always linear and could jump stages. For example, an individual might use a product (acquisition), get tired of the product and contemplate abandoning it (retention), and fall back in love with the product (engagement). Notwithstanding what journey an individual might take, we need to have a game plan for each stage.

 

Customer Lifecycle - Awareness, Acquisition, Engagement, Activation, Retention

2.2.1 Customer Awareness

It all starts with awareness. One study performed by McKinsey & Company indicated that it can take five to eight exposures to an advertisement for it to make people truly aware of a product. People can learn about your product in many different ways such as through traditional marketing, through friends and acquaintances, and through organic news stories. What might be less obvious to the reader is that the product itself can help spread awareness. For example, a social sharing feature can help current customers recommend your product to your friends. Your operations can affect awareness too. For example, your customers can learn about new products that your company is offering when they call customer support.

 

2.2.2 Customer Acquisition

After a person learns about your product, they might decide that they want to try or buy it. This begins the acquisition period. It might seem that acquisition should not be a phase but rather a short point in time, but that is rarely how people become customers. For mobile apps, a new user might have to sign up. That can be a long and complex process involving filling out a registration form, validating their email, and completing their profile. Potential customers can leave the acquisition cycle in any of those steps, and it is not until they get through the whole process before they become active customers.

For other products such as enterprise software, the acquisition process can be extremely drawn out and can involve a sales team that pitches the product, a technical implementation team that helps to answer technical questions and sets up the product, a training team, and so on. The acquisition phase starts with the intent to try or buy a product and ends when the customer actually buys the product or uses the product for the first time.

 

2.2.3 Customer Engagement

Just because a person has used or bought your product does not mean that they are a valuable customer. Even popular apps like Twitter have scores of users that post a few things and then never use it again. Those are not engaged users, and having unengaged users is bad for business. Usually customers fail to become engaged because they either can’t overcome the learning curve or the product fails to meet their expectations. Luckily, it’s possible to systematically and predictably fix both.

Some products have much more compressed engagement cycles. For example, if you are in the market for a luxury yacht, you probably won’t be making a bunch of repeat purchases. However, that does not mean that all yacht owners are equally engaged. Some owners love their luxury yachts, and it shows as they merrily sail around the Mediterranean and extol their yacht’s virtues to their friends. Those are the kinds of customers that you want!

 

2.2.4 Customer Activation

Activation is a big jump in the customer’s level of engagement. The anecdote that I like to tell is my relationship with Amazon. I became an Amazon customer in 2001, when I was a college student. I realized that buying textbooks on Amazon could save me a ton of money, so every term I bought a bunch of books on Amazon. I was a pretty engaged customer. I liked that I could save money, which was tight during my college years, and I was generally happy with the service.

Fast forward a few more years, and Amazon’s offerings broadened. At that point, I was a young professional, and I would occasionally buy a DVD or a book. Once again, I was a solid repeat customer, but I would only buy things that were difficult to track down in a physical store and for which I could wait a week to be delivered. I still preferred to run down to my local BestBuy to buy electronics, DVDs and videogames since I could get it right away.

Fast forward again to 2009 when I went back to graduate school, and Amazon gave me a free Amazon Prime membership since I was a student. That year I bought my textbooks on Amazon, but my purchase behavior started to change radically. Suddenly, I was buying everything on Amazon: DVDs, video games, electronics, clothes, household items, you name it. On reflection, my Amazon Prime subscription annihilated a huge deterrent: having to wait a week for my orders. I didn’t realize at the time what a big roadblock the shipping time was, but now that I had free two-day shipping the floodgates were open. In fact, there was a time when my sons were still babies, when I’d get Amazon shipments multiple times a week filled with diapers, baby clothes, household items. That is what I call Activation!

Activation is a phase in a customer’s lifetime when they become super engaged. Many customers never get to that stage, but that is where the biggest growth can happen.

 

2.2.5 Customer Retention

Unfortunately, all good things must come to an end. Even your most activated customers may get frustrated with your product or company and will choose to leave at some point. More likely, those customers that never got very engaged will abandon your product. The good news is that abandonment does not usually happen instantaneously. There are often signs that a customer is unhappy, and you will likely have ample opportunities to make your customers happy. Even better, you can take steps during customer acquisition and engagement that will prevent your customers from becoming unhappy with your product once they convert to active customers.

I was recently trying a new online project management app. The app and the company behind it did many things right to make sure that I would not get frustrated or disappointed as I became an active user. First, when I signed up for the app, they sent me a personalized email from an individual on their team thanking me for registering and letting me know that I can contact her at any point if I have any questions or issues. Second, when I logged onto the app for the first time, they had a great video tutorial giving me an overview of the features and best practices. On top of that, they had little contextual tutorial messages as I explored the app for the first time.

I set up a simple project to test out the software and invited one of my teammates. It was a pretty solid project management app, but there were a few little things that irked me, and I stopped using it with time. The company saw that I hadn’t used it in a while and they sent me a couple emails (automated, I’m sure, but signed by my customer service representative) asking me if there is anything that they can do to help me with the app. I finally relented and replied to the email and set up a call with my dedicated customer service representative that addressed each of my complaints in turn. She couldn’t solve all of them, but she did help me find other ways to get things done. The company not only did its best to avoid abandonment, but they also successfully retained me when I was not totally happy. I’m still a loyal customer.

 

2.2.6 Engagement Versus Retention

Not everyone will agree with defining retention as simply a measure of keeping customers. To some retention means the act of keeping customers coming back, while engagement is a measure of how intensely they use your product at any one time. For example, they would consider Google Search to be a product with low engagement and high retention given that users use Google Search briefly, but then keep coming back.

Consider a student that is writing a term paper, who uses Google to help them find resources and information, and then never uses Google again. Because they were doing a ton of queries while writing the term paper, we would say that their engagement was high but their retention was low because they never used Google again. I think the majority of people would not consider this person to be highly engaged even if they did submit dozens of searches in a day. Wikipedia users are similar. They use the online encyclopedia intensively but infrequently.

I find it more useful to think of engagement as a measure of how much a customer is using a product, which is a function of intensity and frequency, whereas retention indicates if I am still an active customer. For example, which user is most engaged, one that uses your product once every day or another that uses your product ten times a day every ten days? What about a customer that uses your product twenty times over the span of two days but never touches it again? I would say that the user that used it twenty times and never again is not engaged because he is not an active customer while the other two are roughly equally engaged.

It’s often very ambiguous whether to consider some customers as being active or having abandoned your product, and every company has it’s own cutoff point depending on the nature of your product. For example, many years ago, I signed up for Flickr to share my photos with friends and family but stopped using it after a while and have not logged on for over four years. I still have an account, so am I still an active Flickr customer? It depends on Flickr’s definition of what they consider to be an “active customer.”

 

2.2.7 How Acquisition, Engagement, Retention Work Together

In 2007, a recent Yale graduate, Justin Kan, came up with an idea to stream every minute of his life over the internet. With the help of Emmett Shear, Michael Seibel and Kyle Vogt, he created Justin.tv, which would become the world’s first prominent livecasting service. The team grew Justin.tv to be a fairly solid business employing about twenty-five people, but at some point the founders realized that their idea had hit a ceiling. There were only so many things that it made sense to livecast, and they had tapped out all of them.

Justin.tv rose to about 30 million unique visitors per month, but the majority of their users weren’t super engaged, except for one tiny segment. Emmett Shear made the realization that video gamers, which made up just 3% of their user base, were fanatical users. They would sometimes spend hours streaming themselves playing video games and others would watch them playing for long stretches of time. The four founders decided to focus on addressing just the video game market and created a new version of Justin.tv called Twitch, which became one of the hottest startups in Silicon Valley and sold to Amazon for $970 million.

Justin.tv had the perfect product all along, but they were distributing it to the wrong customer base. Not too many people were super engaged by watching Justin sleep or others sit around in front of the computer, but the video gamers were completely hooked. Once they focused on acquiring the video gamers their product growth exploded. This is one of the best examples of how acquisition, engagement, and retention are related.

Many people forget that the concept of product-market fit is composed of two parts and focus, instead, on just the product. However, it’s not just about building the perfect product. Finding the right customer whose needs your product meets most adeptly is just as important.

If you target the wrong customer, it does not matter how good your product is, you will always have problems engaging those customers because they won’t really have the problem that your product is trying to solve. It’s like selling goose down parkas in Miami Beach. You might have the most stylish, warmest goose down parka in the world, but if you sell it to folks in a place that is constantly hot, your customers won’t love your product and will eventually shove it in their closet or give it away to Goodwill. On the other hand, if you sold that same amazing down parka in St. Paul, Minnesota, your customers would wear it all winter long and would love it for years to come. Their engagement and retention would be off the charts.

To put it simply, if you are effectively targeting and acquiring the right customers for your product, they will appreciate it and will be highly engaged. Customers that are highly engaged, are very unlikely to abandon your product. How much do you like Google? I bet you’re a highly engaged Google user. Are you very likely to stop using Google. Probably not unless another even more amazing search engine comes along.

 

2.2.8 Key Points in the Customer Lifecycle

At the beginning and end of each phase of the conversion funnel are key events that demarcate the end of one and the beginning of another. These points are hugely important because you can influence these events to supercharge your growth.

 

Events Customer Lifecycle

The First Time Someone Hears About or Sees Your Product

A good first impression is trickier than you might think, and it will set the stage for the rest of your relationship with you customer. First, you want to make sure that you’re reaching the right people — those that have the problem that your product is solving. You also want to make sure to communicate how your product is better at meeting their needs or solving their problems. Finally, you should tell them how to get your product or what to do next. Without meeting these three fundamental awareness goals, acquisition becomes an uphill battle. Even engagement, activation and retention can be adversely affected by a poor first impression.

The First Time Someone Uses or Buys Your Product

Usability and customer onboarding make the biggest difference when it comes to creating a great first experience. There is not much that you can do if your product doesn’t have the right features to meet the needs of a customer. Certainly, you can add important missing features with time, but by that time many of the dissatisfied customers will be long gone. Not only that, as soon as you add a new feature, customers will want others, and you will be constantly playing catchup. However, I have found that for a significant set of the customers the right features do exist, but they are hard to find or understand how to use. Moreover, bad usability often forces customers to abandon products long before they learn their full utility. Both of these problems are preventable, and addressing them will go an extremely long way to improving your engagement and retention.

The Customer Uses Your Product More or Buys More

A user will very rarely discover how great your product is in their first use. It often takes many interactions to find out if and how well it will work for the customer. Using or buying a new product is an awful lot like moving to a new city. It takes a while to learn where the good restaurants, parks, and hangout spots are.

Often times we are pleasantly surprised about what the city has to offer. Other times we are disappointed to find that there is not much more to the new place than initially meets the eye. It’s not that the place has changed in the time since you moved; you just have learned it more deeply. Often people discover new things with the help of their friends or colleagues. With your product, you want to be that friend that guides their discovery. Your marketing, product, and operations should be aimed at helping the user discover and learn your product as quickly as possible, so they appreciate it for what it really is.

Customer Becomes a Habitual User or Buyer

The holy grail of growth is the point at which an individual becomes a habitual user or buyer. This is when they enter the activated phase of their customer lifetime. We all are very familiar with this concept. Earlier, I told my story about how Amazon was able to activate me, though free two-day shipping, to become a weekly shopper. If you are on Facebook, you are very likely a habitual user, checking your account at least a few times per week.

On the other hand, there are also many products that I use sporadically. For example, there is a personal finance software that I have been using for the past five years, but I only log onto it when my spending really gets out of hand. According to many reports, there are a great number of Twitter users that are like that. Folks sign up, write a few tweets, follow a few users and then hardly do anything else. On the other hand, there are many Twitter users that post seemingly every minute. The latter are the kinds of customers you want to cultivate with great product experiences, effective marketing, and world-class operations.

Customer Becomes Unhappy with Your Product

Even the best products in the world have unhappy customers. For example, the average customer satisfaction score in the United States is 87%.  That means that about 13% of customers are unhappy at any one time. If your product is a social media app, your customer satisfaction score is likely around 78%.

Disappointment and frustration can be factor of both product experience as well as marketing. You can have the best financial management tool in the world for middle income customers, but if you have high net worth or very poor customers, they will likely become unhappy with your product. On the other hand, you could be reaching the perfect audience, but perhaps your product is difficult to use. This will also not work in your favor.

There are also times when customers simply haven’t been able to figure out how to use your product to truly meet their needs. Luckily, that’s something that you can fix by systematically listening to your customers, experimenting with better ways to help them get the best of your product, measuring the impact that those changes have, and keeping only those that move the needle in the right direction.

Individual Stops Using the Product

Social scientists believe that the last impression is more important than the first. That is why you should make sure to end on a good note as much as you can. If you can do a good job saying goodbye to a customer, you might still get them back in the future. Perhaps they stopped using your product because it was missing a key feature, and when you finally build that feature you want them to try it again! Just as importantly, ending on a good note might mean that they will refer your product to their friends and colleagues. The might say, “It didn’t have this one thing that I really wanted, but it’s still a great product and a caring company!”

 

2.2.9 Marketing, Product, Operations through the Customer Lifecycle

Marketing, product design, and operations can each affect all stages of the customer lifecycle. It’s obvious that marketing can drive awareness, but we also saw with the above examples that product features and even customer support can drive awareness as well as acquisition. Conversely, how good a business is at turning interested folks into customers depends greatly on their ability to market to and draw the right customer. It is important to remember that marketing, product implementation, and operations could potentially affect all parts of the customer lifecycle as we consider how to engineer growth.

 

Marketing, Product, Operations over Customer Lifecycle

Be sure to check back tomorrow to learn about strategizing and prioritizing experiments. New sections of Growthzilla are published every weekday.