3.4 Measuring Non-Digital Marketing Channel Metrics

Some people seemingly default to easy-to-measure marketing channels such as pay-per-click advertisements rather than choosing other channels such as print and television advertisements, whose effectiveness can be more difficult to track. However, you should not let convenience drive you to waste money on less effective channels when, with a little effort, you can also measure, test, and refine more impactful forms of marketing. It’s worth spending some time to discuss a few tactics that will allow you to measure non-digital marketing campaigns.

The key thing to remember when testing different channels is to keep your marketing message the same, so the only thing that you’re varying is that channel itself. Otherwise, you won’t know if the difference in outcomes is due to one channel being more effective than the other, the message resonating more with the audience, or a combination of the two.

Ways to Measure the Effectiveness of Non-Digital Marketing Channels

3.4.1 Simple Survey

There are many opportunities to survey customers, including immediately when they visit your website, open your app, contact your sales team, or complete a purchase. The timing of the survey greatly depends on what you intend to measure. For example, one great way to ascertain what marketing channel most effectively drove new customers is simply to ask them in a survey at the time of purchase.

As mentioned in the first chapter, marketing not only determines the number of prospective customers a product attracts, but also the likelihood of purchase (acquisition) and retention. For example, we could launch an online marketing campaign that falsely exaggerates the capabilities of a product. This campaign could drive lots of prospective customers to our website. Depending on the effectiveness of the website, we will then convert a fraction of these people. Unfortunately, due to the nature of misinformation that brought these customers in, we immediately see large drop-off rates. This example demonstrates three critical stages: initial interest, purchase conversion, and post-purchase retention. We could implement surveys at all three stages to measure how well the marketing campaign drives prospective customers to your product or website, how effective it is at targeting the right customers and conveying accurate information leading to conversion as well as retention.

If the key question that you are trying to answer relates specifically to the effectiveness of driving potential customers to your product then you will want to survey them at the very beginning of the sales cycle when they come to your website, when they contact your sales team, and so on. On the other hand, it is sometimes much more interesting to consider both how well different marketing methods convert interest to actual usage or sales. In this case, you need to survey people after they have used your product, registered for your service, or completed a sale. Finally, it is also useful to have a sense of how well different campaigns attract the right customers and retaining them. In that case, you might want to survey customers some time after they have bought or started using your product. It’s important to acknowledge that customers may not recall the marketing campaign that drove them to try the product, and the data that you get from surveying individuals a long time after they have been exposed to the campaign is likely to be less precise.

Imagine that you are interested in learning how many people that bought your product were driven by your radio advertisement versus an online advertisement. Let’s assume that you are spending $10,000 on online advertising and an equal amount on the radio campaign. After surveying customers post-purchase you find that roughly twice as many people bought your product after hearing the radio advertisement versus those that saw the online advertisement. You compare the two channels based on your survey data and estimate the cost per acquisition as in the table below.

Example Comparison of Radio vs Online Advertising

It seems that the more traditional medium, radio, is a much more effective channel for attracting and converting customers, but how confident should you be in this analysis. You were very careful to keep the messaging the same for both channels, so that should not have affected the difference. But you are concerned that maybe there is a sampling bias where those customers that bought your product after hearing a radio advertisement are more likely to take the survey. This is a legitimate worry, and might have been hugely important if the numbers were closer. However, it is not necessary to be perfectly scientific about this kind of analysis when running growth experiments. It is certainly better to do a more thorough statistical analysis and take greater pains to design a more accurate survey, but such a level of rigor might not be worthwhile in many cases.

3.4.2 Special Landing Pages, Emails, or Phone Numbers

Another very clever way to measure the effectiveness of marketing channels is to tie them to unique website pages, telephone numbers, or even email addresses. For example, you might run a radio advertisement for your product, Sergio’s Widget, and direct folks to navigate to “www.sergioswidget.com/radio.” You might also run a television advertisement which directs people to “www.sergioswidget.com/tv.” The great thing is that you can be quite certain that folks that navigated to the landing page associated with www.sergioswidet.com/radio heard your radio advertisement and not your television advertisement.

This method has some advantages over surveys. First, you do not have to interrupt customers while they are in the process of evaluating or buying your product. Instead, you send them down discrete paths from the very beginning. Second, people might misreport in a survey what piece of marketing led them to discover your product, whereas providing them with a unique link or phone number for each marketing channel forces them to remember more accurately. Third, and most importantly, providing them with a special destination opens up many opportunities to track them throughout the entire acquisition process. In other words, you can track how many people that started at www.sergioswidget.com/radio went on to successfully complete a purchase. This is hugely valuable information.

Tracking marketing channels with unique web addresses and phone numbers also has some disadvantages of which you should be aware. Just as was true with surveys, this measurement technique will give you relative measures of performance. In other words, you might learn which marketing channel is likely to perform best. It’s also very likely that users will navigate to www.sergioswidget.com even though the radio advertisement mentioned www.sergioswidget.com/radio. This might not be a big issue and will not severely skew your relative measures of channel performance if we can assume that this problem occurs uniformly across the various channels. However, if one destination is much easier to remember than the other, you’ll likely have an inaccurate outcome. For example, if the telephone number that you provide in the radio advertisement is 1-800-888-888 while the television advertisement mentions 1-800-289-0973, it will seem that the radio channel is a lot more effective, relatively speaking, than it really is because the phone number is a lot easier to remember. What this all means is that you need to carefully select the unique identifiers that you use so that they don’t end up skewing your results.

3.4.3 Promotional Codes

Another great method for keeping track of acquisition across various marketing channels is providing promotional codes that are specific to each channel. This method is very similar to providing distinct web addresses, phone numbers, or emails, but it is different in the sense that people do not have to go to unique destinations. This can be a big advantage for prospective customers since they don’t have to remember a special web address or phone number. The main disadvantage is that the promotional code must be tied to some kind of discount or reward, otherwise people do not have any incentive to enter it, and this can be costly.

 

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Thursday to learn about evaluation methods that you can use for growth. New sections of Growthzilla are published every week.

3.2 Engagement Metrics for Growth

Unless your business is selling very high-ticket items such as real estate or luxury yachts, which most people buy rather infrequently, you probably want your customers to use your product consistently and intensively. If your customers use your product all the time, it’s a sign that it delivers a lot of value to them, and they will probably keep paying for it or using it. On the other hand, if customers hardly use the product, that signals that your business is providing very little value to them. Perhaps the product is not a great solution for their needs, they are finding it difficult to use your product, or the support that your business provides is very suboptimal and frustrating. In any case, those are serious problems that will probably sink your business in the long-run.

3.2.1 Ratio of People Performing Key Actions

Products exist and thrive in the marketplace because they solve challenges better than other solutions allowing customers to perform actions that are valuable to them. However, such actions vary for each product. For example, a social media app such as Instagram depends on people sharing pictures, reacting to them, and interacting with other. Those responsible for growth at Instagram might be tracking actions such as the number of new posts, the number of replies to posts, the number of likes, the number of times a user shares posts, or even the number of friend invites that an average user sends. On the other hand, if your product is a SaaS project management software, you might want to track actions such as the average number of times that tickets are assigned to team members, the number of project tasks created and closed, or the number of times that team members comment on a task.

3.2.2 Daily, Weekly, Monthly Active Users

Another very useful way to measure engagement of a product is how many people use it in a given span of time. Let’s imagine again that your company has build an online project management tool. What if a 10,000 users created an account, but only a hundred users log in during a given month? The absolute number of registered customers or even customers growth tall you nothing, but measuring how many of those folks use your product in a time period to the total customer base can tell you a lot about how engaging your product is.

The three common measures are daily active users (DAU), weekly active users (WAU) as well as monthly active users (MAU). These metrics capture the total number of users that performed some action, such as signing in or creating a post, that would deem them to be “active.” For example, if you define active users as those that log into your app, and you have 1,000 daily active users, that means that 1,000 users logged into your app during that given day.

The main challenge in this approach is determining what constitutes an “active” user. For example, is an active user one that comes to your app but does not do anything? Or is an active user that comes to your site (or opens your app), logs in, and does something such as creates a task or post? Your numbers will vary greatly depending on how you define “active,” but I would encourage you to be honest with yourself and define active users in a strict enough way that it will gauge engagement.

3.2.3 Ratio of Daily Active Users to Monthly Active Users (Stickiness)

A related metric is the ratio of daily active users over monthly active users, which measures “stickiness” of the product. Let’s consider the extreme cases of this ration to better understand how it signals the engagement of your product. If your product has 1,000 users that are active on any given day as well as 1,000 users that are active in any given month, that implies that the same 1,000 users were active each day for a whole month. In other words, they used your product every day. That is an enviable claim that even the most successful products in the world cannot make.

Consider now a case where you have about 33 daily active users and 1,000 monthly active users. (It’s worth noting that you can’t have less than 33 DAU because they would not add up to 1,000 over thirty days.) Your ratio of daily active users to monthly active users would imply that every day a new set of 33 users were active meaning that none of them used your product on more than one day in a whole month. It would be safe to say that you product is not very engaging.

 

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Tuesday to learn about common retention metrics. New sections of Growthzilla are published every week.

3.1 Acquisition Metrics for Growth

As mentioned previously, the best metrics are ones that fit your business model, objectives, and success criteria. Nonetheless, below are a few acquisition metrics that are commonly used by businesses and can serve as a solid starting point for your own growth efforts. Each of the following metrics can provide a unique perspective on the levers that you can pull to stoke user growth, and it is worthwhile to try incorporating each of them in your process to see which strategic angle might yield the greatest gains. It is also important to note that your growth team can customize and build upon these standard metrics to really capture your unique business and growth models.

3.1.1 Conversion Rate

Probably the most common metric that growth leaders track for online products and apps is conversion rate. In a generalized form, a conversion rate measures the proportion of people that completed a process after starting it. It is important to really pay attention to what events are considered as the “start” and “end” of a process. For example, if you were measuring the conversion rate for registrations, what two events are you actually comparing? Are you comparing the number of people that successfully submitted a registration form to those all those that came to your website or just to those that came to the registration form? Or maybe then successful action is not just submitting the registration form but rather also being approved. Some common conversion rates are the proportion of people that:

  • Saw an online advertisement and navigated to a website,
  • Saw a call to action and navigated to a high-value page or screen,
  • Started the registration process and successfully finished it, and
  • Added an item to their cart and successfully purchased that item.

Some conversion rates can measure the effectiveness of your acquisition efforts as well as engagement, so it is important to consider what actions you are tracking and how they relate to business functions such as marketing, product development, and operations. For example, the conversion rate of people that started a registration process and successfully finished it might depend mostly on how the registration process is implemented rather than on the marketing that drove people to the registration page.

Conversion Rate Formula

At the same time, it’s very likely that the conversion rate for the registration process is also a function of the effectiveness of the marketing efforts. Let’s imagine two scenarios for an online project management software. In one scenario, marketing does a great job communicating the benefits of the tool to prospective customers. In another scenario, marketing does not communicate the value proposition very well. It’s reasonable to assume that those individuals that were exposed to the more effective marketing campaign would be more likely to complete the registration process because they have a greater conviction that the project management tool will be valuable in their work.

Clearly, the conversion rate for the registration process is probably a function of both product implementation (how easy it is to register) and marketing effectiveness (how convincing is the marketing messaging). Other conversion rates such as the proportion of people that see and click on online advertisement can capture the effectiveness of your marketing message and channel alone. The important point to remember is that if the conversion rate you pick could be affected by more than one function (marketing, product implementation, operations), your team should carry out your experiments on one area at a time to isolate and optimize their effects. For example, if you believe that the registration rate is affected by both how convincing the marketing message is as well as how easy it is to actually register, you would likely want to try optimizing marketing and product implementation separately rather than at once.

One question that business leaders typically have is what is a “good” conversion rate? It’s impossible to provide a general benchmark because conversion rates vary not only by product type but also by what you are trying to measure. For example, the conversion rates for seeing and clicking on online advertisements tend to be in the single digits while conversion rates for registration are likely much higher. In fact, it is really not useful to compare your conversion rates to benchmarks. Rather, realize that they can always be improved, and aim to constantly be ratcheting them up.

Another important point to understand is that conversion rates in themselves do not paint a holistic picture of growth performance nor do they provide insight to all the opportunities that exist. It is important to look at other metrics such as the customer acquisition cost and the rate of visits to key pages to gain a broader perspective on optimization possibilities.

3.1.2 Customer Acquisition Cost (CAC)

Another common metric used to gauge the effectiveness of a company’s acquisition efforts is customer acquisition cost (CAC), which is primarily an indicator of the efficacy of the company’s marketing efforts. As an example, imagine that you are using two channels to advertise your product, pay-per-click as well as online video advertisements. You do the numbers and find that the video advertisements have a conversion rate of five percent while the pay-per-click advertisements have a conversion rate of only two percent. If we based our growth engineering solely on conversion rate, this would be an open-and-closed case. Clearly, video wins, right?

It’s not necessarily true that video is the better marketing channel. Simply put, it could be that video costs way too much even though the conversion rate is much better. The video production costs and the ad placement costs can add up to the point that what you are paying for an actual sale is inferior to what you would have paid had you advertised with a pay-per-click ad. Even worse, it could be that you are losing money on every new customer because your cost per customer acquisition is less than the average revenue for that customer.

Customer Acquisition Costs Scenarios

Let’s consider an example, since it may not be intuitive how marketing with a higher conversion rate can be less worthwhile than ones with a lover conversion rate. Imagine that you are marketing a project management software over two channels: online video and pay-per-click (see table above). Which one is more effective?

For your video campaign, you had to hire an advertising agency to make the video, which cost you a total $50,000. Over the course of your the campaign, the video advertisement was displayed 100,000 times, and it cost you one cent every time it was displayed (cost per impression) for a total of $1,000. Of the 100,000 times that your video was displayed, people clicked on the link in the advertisement 5,000 times, which gives you a click conversion rate of 5%. Taking into account the production costs ($50,000) and the placement costs ($1,000), you have spent $51,000 to get 5,000 people to click on the advertisement and navigate to your site. Your cost per conversion is $5.10 assuming that we measure conversions as the proportion of people that click the link in the video and navigate to your site.

Let’s now consider your pay-per-click campaign. You have experience running pay-per-click advertisements, so you create the campaign on your own without having to hire expensive outside help, which makes your production costs zero. Just like the video campaign, your advertisement appears 100,000 times and gets 2,000 clicks for a 2% click conversion rate. In addition, it costs you $1.00 every time someone clicks the link in your advertisement and navigates to your site, which adds up to $2,000. Your net cost per conversion is simply the cost per click or $1.00. That means that even though the conversion rate on the video ads was twice as high as that for the cost-per-click campaign (5% vs 2%), the cost per conversion on the cost-per-click is less than one-fourth of video ($1.00 vs $5.10).

Should you invest your money in the pay-per-click campaign? Remember that customer acquisition cost is measure how much marketing spend it takes to get a converted customer. Clicking on an advertisement does not mean that individual will become a registered or paying customer. Let’s factor this into our fictitious example.

Let’s say that you are tracking customers that come from your video vs. pay-per-click campaigns. You find that for every two people that came from the video advertisement, one actually buys a subscription to your software whereas only one in four converts to an actual sale if they come via the pay-per-click advertising. Taking into account the conversion rate between a click and a purchase, you get a customer acquisition cost from video to be $10.20 versus $4.00 for the pay-per-click campaign. The campaign with the lower click conversion rate is actually more efficient! In fact, if the net revenue per customer is $10.00, you would be losing money on the video campaign. That is why conversion rate is extremely useful, but cost per acquisition is also critical in deciding which messaging and channel are optimal. Having established the conversion rates and customer acquisition costs for your marketing campaigns, you might want to set growth goals for your customer base. A great metric to track that goal is the rate of new customer acquisition, which is covered next.

3.1.3 Rate of New Customer Acquisition

You have two thousand paying customers. Is your company doing well in acquiring new customers? It’s difficult to tell. Perhaps your business had 1,900 customers a year ago and the customer base has barely grown. The raw number of customers does not necessarily provide a ton of insights. A much more useful measure is the rate of new customer acquisition, which tells you how well your acquisition efforts are performing in a given time period.

Customer Acquisition Rate Formula

By comparing customer acquisition rates you can tell if your business is acquiring more users now versus another historical time period. Everyone wants “hockey stick growth.” That is simply an accelerating growth rate, which means that you want your customer acquisition rate to be increasing with time. What is even more brilliant is that the customer acquisition rate can be combined with the churn rate, which is a measure of how many customers are leaving in a given time period, to provide a net customer growth rate. In other words, you want your customer acquisition rate to be higher than your churn rate, which is a measure of the effectiveness of your engagement and retention efforts.

The customer acquisition rate is a powerful top-line metric, since it measures what you ultimately want to increase: customer growth. While it is important to also track metrics that give you deeper or more specific insights into parts of your growth efforts, your team should always be tracking customer acquisition rate since not matter what the other numbers say, they better be collectively pushing the customer acquisition rate up.

3.1.4 Rate of Visits to Key Pages or Screens

Another common metric to track for online products and mobile apps is the rate of visits to key pages or screens as compared to overall visits to the site or user sessions on an app. Some businesses make the mistake of tracking just top-line metrics such as the customer acquisition rate explained above. This is a problem because actual purchases are preceded by many interactions and steps that contribute in aggregate to the overall acquisition or purchase rate. Understanding how customers progress through each step in the overall process allows your team to optimize the constituent parts of the full flow. For example, your team might be interested in the rate of visits to the pricing page to see if improvements in the navigation has resulted in more potential customers getting to the point where they evaluate the cost of the product or service.

By combining broad metrics such as customer acquisition costs and customer acquisition rate with specific and targeted metrics such as various conversion rates and visit rates to key pages and screens, your team will be equipped with data that can inform small changes that drive your progress toward your overall charted course.

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back next Tuesday to learn about common engagement metrics. New sections of Growthzilla are published every week.

Chapter 3: Tracking the Right Growth Metrics

3.0 Good Growth Metrics

Without good data, you won’t be able to reliably analyze the results of your growth experiments, and you’ll be back to a guessing game. Assuming that your data collection methods are sound, you still have to ensure that you are capturing the right metrics — ones that can help you understand whether an experiment has met its objective as well as your success criteria. In this chapter, we will review the characteristics of good growth metrics, and some standard metrics that can get your team started.

There are many qualities that make metrics “good,” but key characteristics to look for are the following:

  • Captures your objective
  • Easy to measure
  • Accurate
  • Actionable
  • Predictive
  • Based on your growth model

Captures Your Objective

Some might claim that it’s intuitive which metrics are appropriate for certain experiments, but this is not true. Experiments are meant to meet certain objectives and satisfy success criteria that you set. For example, there is often a tension in trying to optimize customer acquisition. Should you focus on quantity or quality? You can cast a bigger net to get more customers, but those customer probably won’t be as engaged as customers that you could get through more targeted acquisition efforts.

If your objective is just to grow the raw number of customers, you might track metrics such as the rate of new registrations or the conversion rate, which captures the proportion of people that finish the registration process to the number of people that start it. However, if your objective is to build a customer base of very engaged customers, you might prefer to measure things such as your daily active users, which captures how many of your users spend time with your product each day. Another metric that might be appropriate for capturing your objective to build an engaged customer base might be the average time that a customer is using your product in a given week. As you can see, the metrics that are appropriate for each objective are as disparate as the objectives themselves, so it’s important to take the objective into account when picking a metric to capture.

Easy to Measure

With the digital age and the proliferation of tools, many metrics have become easy to capture. However, some metrics remain difficult to capture. Those data points tend to relate to the non-digital realm such as measuring the acquisition rate of customers that saw a billboard advertisement or the retention rate of people that saw a television advertisement. Often times, it’s possible to measure the effects of a given experiment using a number of different metrics, and it’s worthwhile to focus on those metrics that are easier to measure to ensure that you do not waste effort by running experiments without good data outcomes. However, it’s important to understand that there is a danger of defaulting to those metrics that are easiest to garner and not those that provide the best basis for analysis. You will need to determine the right balance for your goals. Sometimes it is preferable to opt for difficult to measure data if it will give you a clearer picture of whether or not an experiment successfully met your objective.

Accurate

Good metrics should also be strongly tied to outcomes rather than loosely represent them. For example, let’s say that you wish to impact how many purchases customers make. You could choose to measure either the customer’s intent-to-purchase or actual purchases. It is well known that intent to purchase is not always an accurate indicator of actual purchases. Therefore, you would likely be better off measuring the actual volume of purchases rather than a virtual metric such as intent.

Actionable

All metrics should be actionable in theory, but some are definitely easier to act upon than others. How actionable metrics are greatly depends on how specific they are. Let’s say that your product is an online, subscription-based task management tool. If I told you that the average customer satisfaction score for your product is 6.9 out of 10, would you know immediately what to do next? What if I told you that 4 out of five 5 customers that register for your product start but never finish creating their first task entry? Would you have a better idea of where to focus and what to try next? I hope that the second metric would arm you with much more actionable data. It takes a lot of time and resources to conceive and implement an experiment, so it’s well worth it to give some thought to how actionable the data that you intend to measure will be.

Predictive

It is always a good idea to track data that gives you insight into your business and are outside the context of a particular experiment. These metrics are different in nature than those that you should be tracking to identify if a particular growth experiment has met your objective or success criteria. Instead, the data should help to shape your growth strategy by helping you anticipate problems before they become hugely disruptive to your business. A classic example of this is the distinction between the rate of customer service emails and customer churn rate, which indicates how quickly existing customers are abandoning your product.

Let’s go back to the example of an online task management software. Imagine that your team changes a big piece of your product, such as, how tasks are created by users. Should you be tracking the customer churn rate or the rate at which your customers are contacting customer support. The former is usually calculated after the fact, whereas you can see a spike in customer support inquiries immediately. Therefore, the rate at which your customers are contacting customer support will let you react much more quickly than the churn rate, so you should be keeping your eyes on customer support emails or calls. As you think about what metrics you should track to give you general insight into the health of your business try to think about what data can tip you off about problems earliest.

Based on Your Growth Model

Although metrics such as daily active users or conversion rates are fine, nothing beats creating a data capture strategy based on your own growth model. Doing so ensures that you have precisely those metrics that are tied to variables that most directly affect the growth rate of your business. Your revenue and customer growth models should tell you precisely what metrics your team should be tracking.

Marketplace Revenue Growth Model

Let’s consider an online marketplace with the above model as an example. At first glance, you can see that you should be capturing the following metrics:

  • The number of sellers
  • The average number of posts per seller per time period
  • The number of views of items for sale
  • The number of buyers
  • The number of times an average buyer visits the marketplace in a given time period
  • The ratio of the number of times that buyers visit an item page to the number of completed sales
  • The average fee that your company charges per transaction
  • The average sale price of items in the marketplace

Not only did the revenue model help us identify novel metrics that are perfectly suited for an online marketplace, such as the number of views of items for sale, but it also helped us make standard metrics, such as conversion rate, specific and contextual to the business.

As mentioned above, making changes to marketing, product implementation, and operations is resource intensive, so it’s well worth spending some time to identify metrics that will help your team clearly determine whether your growth experiments meet their objective and satisfy success criteria to ensure that you are not wasting time and resources. Of course, it would be ideal if all of your metrics satisfied all of the criteria above, but the reality is that many will not. In that case, it’s best to focus on ensuring that the metrics that your team chooses to capture align with your model and are accurate. Beyond your tailored metrics, there are others that have become standard practice to track. We will explore these in greater details in the following sections.

This post is part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018. Be sure to check back on tomorrow to learn about common acquisition metrics. New sections of Growthzilla are published every weekday.

 

2.4 Running Successful Growth Experiments

Sample Growth Objectives and Experiments

This post is a part of the Growthzilla Book series, which is an online draft of the print edition that will be available in 2018.

By utilizing experimentation to help us decide if changes to product, marketing, and operations are effective, we avoid having to rely solely on our intuition. Without experimentation, we would implement changes and hope that we’re right slightly more than half of the time. Rather than engineering growth, we would be relying on an art form, which would be dominated by a select few that had outstanding intuition such as Steve Jobs (or those who claim to have this level of intuition). Growth engineering would be inaccessible to the majority of us.

Experiments make growth accessible to nearly everyone because they follow systematic ways of testing hypotheses to reach specific outcomes and are not unlike experiments in the physical and social sciences. Admittedly, growth experiments are usually not as rigorous as in academia, but the fundamentals are still the same. Anyone that learns the basic experimentation methodology can lead successful growth development at their company. Of course, you will likely be more effective with greater experience and practice, but it’s important to learn strong fundamentals from the beginning.

In this section, we will review the key components of successful experimentation. The first step is understanding what are you trying to achieve with your experiment. Are you trying to improve how long users spend on your site or how quickly the can get their tasks done? Then we will consider how to measure whether or not the changes that you implement have successfully accomplished that objective, or not. Then we will discuss ways to create a sounds hypothesis about ways to reach your objective. Finally, we’ll review the nuts and bolts steps in running good experiments as well as gathering and analyzing results.

Read more