Formulating A Hypothesis - Designing Marketing Experiments - Part 1 of 4

The job of any marketing department is to develop effective ways of reaching customers.

In order to do that effectively, however, you must first figure out who your customers are and what they like. You may have learned the standard tactics: upselling, cross-selling, targeted messages, discounts, loss-leaders, and so on. What you haven’t necessarily learned is how best to apply these tactics to your brand and customers. In order to identify and refine an effective marketing strategy, you have to find ways to test it. Once you have identified what tactics work best, you can also use controlled experiments to determine how well a given tactic works, and then develop ways to improve it further.

The benefits of running controlled marketing experiments can be direct and tangible. This can help marketing departments stand out in companies where many people (the CEO included) have only a vague notion of what marketing has to do with the brand, much less how it contributes to the bottom line. By running controlled experiments, marketers can figure out what strategies work, measure their impact on profits, and deliver consistent results. Over the next few weeks, we’ll be discussing how to design a marketing experiment and make sense of the results. We’ll begin our series with how to create a well-defined hypothesis.

Imagine for a moment that we’re the marketing department at a large company. We’re developing a new marketing strategy with the goal of improving customer retention. One of our ideas is to include an email to our customers, perhaps with a discount of some kind. Our boss, who is not the savviest of technology users, says he doesn’t think that customers would respond to emails, and instead wants to go with an expensive direct-mail advert. One way we might convince our boss to join the digital age is by designing an experiment to gauge the effect of email marketing on sales. At the heart of our experiment is our hypothesis. We can start with something simple, such as:

Email marketing increases sales.

Notice that our hypothesis takes the same form as the conclusion we are trying to prove. [ref] Technically, we are trying to disprove our hypothesis, and after sufficient failure to do so, we accept it to be true. This subtlety, while interesting, is not especially relevant for our purposes.[/ref]

Our particular hypothesis describes a cause and effect relationship, as in, “Action A leads to Result B.” We could formulate our hypothesis to test other types of relationships, but for now we’ll still with cause and effect.

At the moment however, both our cause and effect are only vaguely defined. A good hypothesis has to be specific enough to actually test, and it would take a massive number of experiments to determine if all email marketing increases sales. Furthermore, “sales” is itself a rather difficult objective to quantify. Right now we haven’t defined either a time frame or a target audience, which will make it difficult to effectively measure how effective our email marketing has or hasn’t been.

In order to create a useful and manageable experiment, we need to narrow our focus. Rather than testing “email marketing,” let’s test something more concrete, such as “Emailing customers a 20% discount.” And rather than looking for an increase in sales, we’ll look to see if customers who received the discount made a purchase sometime during the following week. Our revised hypothesis might look something like:

Emailing customers a 20% discount increases the likelihood that they will make a purchase in the following week.

Now we have a well-formed hypothesis that is specific enough to test. Our next step will be to test this hypothesis and quantify how much more likely a customer is to make a purchase after receiving our discount email. In our next post, we will discuss how to set up the proper control groups and make these measurements.

Every customer has a story. Make the most of it.

Previous Article
A Bayesian Approach to A/B Testing
A Bayesian Approach to A/B Testing

Next Article
How to Make "Average Order Value" a Valuable Metric
How to Make "Average Order Value" a Valuable Metric