My product management toolkit (28): testing price sensitivity

Normally when I talk to other product managers about product pricing, I get slightly frightened looks in return. “Does that mean I need to set the price!?” or “am I now responsible for the commercial side of things too!?” are just some of the questions I’ve had thrown at me in the past.

“No” is the answer. I strongly believe that as product managers we run the risk of being all things to all people — see my previous post about “Product Janitors” — and I therefore believe that product people shouldn’t set prices. However, I do believe it’s critical for product people to think about pricing right from the beginning:

  • Do people want the product?
  • Why do they want it?
  • How much are they willing pay for it?

Answers to these questions will not only affect what product is built and how it’s built, but also how it will be launched and positioned within the market. I’ve made the mistake before of not getting involved in pricing at all or too late. As a result, I felt that I was playing catchup to fully understand the product’s value proposition and customers’ appetite for it.

Fortunately, there are two tools I’ve come across which I’ve found very helpful in terms of my comprehending the value a product is looking to achieve — both from a business and customer perspective: the Van Westendorp Pricing Sensitivity Meter and the Conjoint Analysis respectively.

The Van Westendorp Pricing Sensitivity Meter has helped me to learn about the kinds of pricing-relating customers to ask (target) customers:

  • At what price would you consider the product to be so expensive that you would not consider buying it? (Too expensive)
  • At what price would you consider the product to be priced so low that you would feel the quality couldn’t be very good? (Too cheap)
  • At what price would you consider the product starting to get expensive, so that it is not out of the question, but you would have to give some thought to buying it? (Expensive/High Side)
  • At what price would you consider the product to be a bargain — a great buy for the money? (Cheap/Good Value)

The aforementioned Van Westendorp questions are a good example of a so-called “direct pricing technique”, where the pricing research is underpinned by the assumption that people have a basic understanding of what a product is worth. In essence, this line of questioning comes down to asking “how much would you pay for this (product or service)?” Whilst this isn’t necessarily the best question to ask in a customer interview, it’s a nice and direct way to learn about how customers feel about pricing.

Example customer responses to the Van Westdorp questions — Taken from: http://www.5circles.com/van-westendorp-pricing-the-price-sensitivity-meter/

The insights from applying these direct questions will help in better understanding price points. The Van Westendorp method identifies four different price definitions:

Point of marginal cheapness (‘PMC’) — At the point of marginal cheapness, more sales volume would be lost than gained due to customers perceiving the product as a bargain and doubting its quality.

Point of marginal expensiveness (‘PME’) — This is a price point above which the product is deemed too expensive for the perceived value customers get from it.

Optimum price point (‘OPP’) — The price point at which the number of potential customers who view the product as either too expensive or too cheap is at a minimum. At this point, the number of persons who would possibly consider purchasing the product is at a maximum.

Indifference price point (‘IPP’) —Point at which the same percentage of customers feel that the product is getting too expensive as those who feel it is at a bargain price. This is the point at which most customers are indifferent to the price of a product.

Range of acceptable pricing (‘RAI’) — This range sits between the aforementioned points of marginal cheapness and marginal expensiveness. In other words, consumers are considered likely to pay a price within this range.

Van Westendorp price sensitivity meter (example) — Taken from: https://www.qualtrics.com/uk/market-research/pricing-research/

 

In addition to the Van Westendorp Price Sensitivity Meter, I’ve also used Conjoint Analysis to understand more about pricing. Unlike the Van Westendorp approach, the conjoint analysis is an indirect pricing technique which means that price is combined with other attributes such as size or brand. Consumers’ price sensitivity is then derived from the results of the analysis.

Sample conjoint analysis question — Taken from: https://www.questionpro.com/survey-templates/conjoint-analysis-retirement-housing/
Sample conjoint analysis question — Taken from: https://www.questionpro.com/survey-templates/conjoint-analysis-retirement-housing/

 

When designing a conjoint analysis study, the first step is take a product and break it down into its individual parts. For example, we could take a car and create combinations of its different parts to learn about combinations that customers prefer. For example:

Which of these cars would you prefer?

Option: 1

Brand: Volvo

Seats: 5

Price: £65,000

Option: 2

Brand: SsangYyong

Seats: 5

Price: £20,000

Option: 3

Brand: Toyota

Seats: 7

Price: £45,000

This is an overly simplified and totally fictitious example, but hopefully gives you a better idea of how a conjoint analysis takes into account multiple factors and will give you insight into how much consumers are willing to pay for a certain combination of features.

Main learning point: I personally don’t expect product managers to set prices for their products or design price research. However, I do think we as product managers benefits from a better understanding of the pricing model for our products and a better understanding of what constitutes ‘value for money’ for our customers. The Van Westendorp Price Sensitivity Meter and the Conjoint Analysis are just two ways of testing price sensitivity, but are in my view to good places to get started if you wish to get a better handle on pricing.

Related links for further learning:

  1. Van Westendorp pricing (the Price Sensitivity Meter) – 5 Circles Research
  2. Conjoint analysis – Wikipedia
  3. Why You Should (Almost) Never Use the van Westendorp Pricing Model
  4. Van Westendorp’s Price Sensitivity Meter – Wikipedia
  5. Pricing research: A new take on the Van Westendorp model | Articles | Quirks.com
  6. Easy Guide: How To Run a Van Westendorp Pricing Analysis – Dimitry Apollonsky
  7. Van Westendorp Price Sensitivity Meter
  8. Conjoint Analysis – introduction and principles

 

Book review: Sprint (Part 2 – Day 1)

In my previous post I started looking at doing 5-day sprints to discover and test solutions for a problem that you’re trying to solve. This follows my reading of “Sprint” by Jake Knapp, John Zeratsky and Braden Kowitz. Once you’ve set the stage for a sprint, it’s time to kick things off: the first day of a sprint is all about agreeing on the challenge that you’re looking to have tackled by the end of the sprint. On the Monday, i.e. the first day of the sprint, the focus is on the following activities: (1) agreeing on a long-term goal (2) making a map of the challenge (3) asking experts and (4) picking a target.

Agree on a long-term goal (‘start at the end’)

You start the sprint by asking the the team “why”, make sure everyone is on the same page about what we’re trying to achieve. Why do we want to create this product? Why are we doing this project? Why do we want to solve this problem? Where do we want to be in 3 months, 6 months, 1 year, even 5 years from now and why? What will success look like? Agreeing on a long term will bring the answers together in a shared purpose.

Once you’ve got a shared understanding of the underlying “why” and have set a long-term goal, you come up with number of specific sprint questions, which you can derive from the assumptions and questions that the team might have. To get the team thinking about some of these questions, you can use the following prompts:

  • What questions do we want to ask in this sprint and why?
  • How will we subsequently utilise the answers to these sprint questions and outcomes?
  • To meet our long-term goal, what has to be true?
  • Imagine we travel into the future and our product or project failed. What might have caused that failure? How can we best mitigate this risk?
  • To reach customers for this product, what has to be true?
  • To deliver value to these customers, what has to be true?

Fig. 1 – Sample long term goal and sprint questions:

Long term goal: More people buying snacks online.

Sprint questions:

  • Are people looking to buy snacks online?
  • What is the experience customers are looking for when buying snacks online?

Map the challenge

Challange Map

Fig. 2 – Example of a Challenge Map: Flatiron Health’s clinical trial enrolment map – Taken from: https://zapier.com/blog/google-ventures-design-sprint/

Creating a map is a great way to understand the steps the customer has to go through to achieve a desired outcome (see a good example in Fig. 2 above). Each map is customer-centric, with a list of key actors on the left. Each map is a story, with a beginning, a middle and an end. These are the common elements of a Challenge Map:

  1. List the actors (on the left) – The “actors” are all the important characters in your story. Most often, they’re different kinds of customers.
  2. Write the ending (on the right) – Write the outcome that the customer wants to achieve.
  3. Words and arrows in between – There’s no need for any fancy drawings; the map should be functional, and simple boxes and arrows should suffice.
  4. Keep it simple – Your map should have from five to around fifteen steps. If there are more than twenty, your map is probably too complicated.
  5. Ask for help – As you create the map, you should keep asking the team, “Does this map look right?” or “What are we missing?”

Ask the Experts

Nobody knows everything and it’s therefore critical that you engage with a range of ‘experts’. One of the biggest challenges of running a sprint is that you’ve got to gather a lot of information and make sense of it in a relatively short space of time. Having short conversations – approx. 30 minutes per conversation – with experts will help massively in collating relevant detail quickly.

Pick a Target

Selecting one target customer and one target event is the final activity of the first day of the sprint. The Decider needs to decide on the target customer and the customer event. Whatever she chooses will become the focus of the rest of the sprint – the sketches, prototype, and customer interviews all flow from this decision. Naturally, this can be a group decision, but it helps to assign decision-making responsibility to a single person.

Once you’ve selected a target, take a look back at your sprint questions. You usually can’t answer all those questions in one sprint, but one or more should line up with the target.

Main learning point: The first day of the sprint should really lay the groundwork for the rest of the sprint. Avoid the temptation to dive straight into solutions. Instead, spend the first day of the sprint to agree on a long-term goal and selecting a specific target to focus on!

Related links for further learning:

  1. http://www.nirandfar.com/2016/03/good-products-start-good-questions.html
  2. http://blog.invisionapp.com/inside-design-google-ventures/
  3. https://stfalcon.com/en/blog/post/how-to-quickly-check-startup-ideas-with-sprint

My product management toolkit (3) – Goal setting

As part of my product management toolkit, I’ve thus far covered the creation of a product vision and the definition of a product strategy. The next thing to look at is goal setting: what are the business goals that a product strategy and or roadmap need to align with? I’ve learned the importance of goals to help define or assess a product strategy. I would even go as far as saying that if your product strategy, roadmap, backlog or – low and behold – your actual product don’t align with your business goals, you’re setting yourself up for failure.

Tool 3 – Goal Setting

What are goals? – This is what Wikipedia has to say about “goals”: “A goal is a desired result that a person or a system envisions, plans and commits to achieve; a personal or organizational desired end-point in some sort of assumed development. Many people endeavour to reach goals within a finite time by setting deadlines.” In other words, what is it the that we are looking to achieve, why and by when?

I typically look at goals from either of the following two angles: metrics or ‘objectives-key-results’ (‘OKR’s). From a metric perspective; what is the single metric that we’re looking to move the needle on, why and and by when? What does this impact look like and how can we measure it? For example, a key business goal can be to increase Customer Lifetime Value with 1% by June 2016. To be clear, a metric in itself isn’t a goal, the change that you want to see in metric is a goal.

From an OKR perspective, the idea is to outline a number of tangible results against a set, high level, objective. For example:

Objective: To enable sellers on our marketplace platform to make business and product decisions based on their sales and performance data generated from their activities on our platform.

Result 1: Our sellers making key business and product decisions before and throughout Christmas 2015

Result 2: Our sellers can look at their historic sales data so that they’ve got more sales context for their decision-making

Typically, there will be a set of overarching business goals that have been established and our responsibility as product managers is to link our product goals to these objectives, so that our product strategy is fully aligned with the business strategy.

okr-deck

Taken from: http://www.businessinsider.com/googles-ranking-system-okr-2014-1?IR=T

What goals aren’t – Goals aren’t a strategy or specific features. This might sound obvious, but often see cases where people do confuse things; setting goals without a strategy to achieve them or having a roadmap that doesn’t align with business goals.

In contrast, the point of a strategy or a roadmap is to highlight the ‘how’, the steps that need to be taken to achieve specific goals.

When to create goals? – It’s simple: if you join an organisation and hear “we don’t have business goals”, you know what to do! My point here is that a product strategy or roadmap that isn’t aligned with broader business goals, is just a loose collection of features or random solutions. The one thing to add is that some early stage startups tend to get really hung up on a whole range of specific goals or metrics. I’d always recommend to keep it simple and focus on a single goal or metric, understand what your (target) users’ needs are and how are they actually using your product.

Characteristics of good goals – I can imagine that a lot of you will have a heard of SMART goals:

  • S = specific, significant, stretching
  • M = measurable, meaningful, motivational
  • A = agreed upon, attainable, achievable, acceptable, action-oriented
  • R = realistic, relevant, reasonable, rewarding, results-oriented
  • T = time-based, time-bound, timely, tangible, trackable

 

SMART 2Example taken from: http://business.lovetoknow.com/wiki/Examples_of_SMART_Goals_and_Objectives

Main learning point: In my view, setting and understanding goals is just important as creating a strategy to achieve them. Before I delve into creating a product strategy or roadmap, I’ll always try to make sure I fully understand the business objectives and translate those into specific, measurable product goals.

 

Related links for further learning:

  1. https://medium.com/@joshelman/the-only-metric-that-matters-ab24a585b5ea#.z74gt29wa
  2. https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/
  3. http://www.theokrguide.com/
  4. http://www.businessinsider.com/googles-ranking-system-okr-2014-1?IR=T
  5. http://www.producttalk.org/2014/01/how-to-set-goals-that-drive-product-success/
  6. http://www.kaushik.net/avinash/web-analytics-101-definitions-goals-metrics-kpis-dimensions-targets/
  7. https://marcabraham.wordpress.com/2013/05/03/book-review-lean-analytics/
  8. http://www.slideshare.net/abrahammarc1/product-roadmaps-tips-on-how-to-create-and-manage-roadmaps
  9. https://www.projectsmart.co.uk/smart-goals.php

No more features on product roadmaps – Have themes or goals instead!

You might have read my one of my previous blog posts about the so-called goal oriented roadmap. I  prefer goal-oriented roadmaps over their more traditional counterparts. My problem is that ‘classic’ roadmaps contain a mix of features and timings, but don’t provide any context whatsoever (I’ve included an example in Fig. 1 below). I typically don’t include features on a roadmap and focus on user or business problems instead. As such, the roadmap becomes a tool for ‘working backwards’ – enabling product teams to explore solutions for given problems or desired results.

I recently learnt from Jared Spool and Bruce McCarthy about adding “themes” to product roadmaps. McCarthy told Spool about the concept of adding themes, and Spool became an instant fan. Like me, McCarthy isn’t a fan of having features on a roadmap. Instead, he suggests adding “themes” to the roadmap and this is why:

  1. Open to ‘options’ –  McCarthy believes that by having specific features on a roadmap, you run the risk of sending a product team down a certain avenue (and closing off any potential side streets). For example, if you were to just put “data dashboard” on a roadmap, chances are you’ll end up with a data dashboard. However, what would happen if you were to put “provide our customers with data to make key decisions” or “no data access” on the roadmap instead? By focusing solely on solution you run the risk of falling victim to what Josh Wexler calls solution sickness. Solution sickness is all about fixating on a solution and ignore any alternative ways of solving a problem.
  2. Customer focus – Jared Spool makes a great point when he says “When companies talk about features, they are saying, “Look at us. Look at what we can do.” He goes on to explain that “When companies talk about the problems of customers, they are saying, “Look at what you’re dealing with. Look at how we want to help.” It’s very easy to get fixated on features and forget about the underlying problems you’re looking to solve.
  3. Helping trade-off decisions – One of the things I love about both the goal-oriented and ‘theme’ approach to roadmapping is that it forces you to consider ‘why’ you want to develop certain products or services. What problem are we looking to solve and why? Why do we feel this problem is worth solving? Why should we prioritise this problem over others? Both the goal oriented and the theme roadmap approaches help make customers and their problems the core of everything you do as a product person. When I use the goal oriented roadmap, I always take out the “feature” layer purely because I don’t want us to be pinned down to one solution.
  4. “Marketing the story of solutions” – Which do you think would be an easier story to sell to customers? Option 1: “You can now use this real-time data dashboard which you can access via your content management system, enabling you to filter the data and extract reports” or Option 2: “You can use our data to decide on whether now is good time to sell your house?” … Easy, isn’t it!? As Jared Spool points out, the ‘theme’ approach makes it much easier for marketing teams to tell a story about a product or service: “Here are the problems we set out to solve and here’s how we solved them.”
  5. A more cohesive design – Design is another area which is positively impacted by a shift in focus from solutions to problems. From experience, I know how much easier it is to prioritise against a problem that you’re looking to solve, instead of trying to cram in a lot of features which you think might have an impact. As Spool puts it: ” Without a commitment to specific solutions, the team has flexibility.” As a result, product teams stand a better chance of creating simpler, well designed products.

Main learning point: I really encourage all product managers – the ones who aren’t doing so already – to think much more in terms of user problems instead of focusing on features. Not only will this help you in focusing your product efforts, I believe it will also make you much more customer centric.

 

Fig. 1 – Example of a traditional roadmap – Taken from: https://roadmunk.com/

project-roadmap

 

Fig. 2 – Goal oriented roadmap example – Taken from: http://www.romanpichler.com/blog/agile-product-roadmap/

SampleGOProductRoadmap_DanceGame

Related links for further learning:

  1. https://medium.com/@josh_wexler/beware-solution-sickness-producttank-nyc-talk-caffc23becfa
  2. https://medium.com/uie-brain-sparks/themes-a-small-change-to-product-roadmaps-with-large-effects-a9a9a496b800

Gathering meaningful data during the user journey

Since I started looking into omni-channel metrics last year, I’ve been learning how to best gather meaningful data at each step of the user journey. I recently came across a great piece by Gary Angel titled “A Data Model for the User Journey”. In his article, Gary aims to address the multi-source nature of our data touchpoints, and the issues brought about by the differences in the level and type of detail data. He rightly points out that these differences in data make any kind of meaningful analysis of the user journey virtually impossible. Gary provides a number of useful steps to tackle this problem:

  1. Create a level of abstraction – Gary first suggestion is to get to a level of abstraction where each data touchpoint can be represented equally. One way of doing this is to apply Gary’s “2-tiered segmentation” model. In a 2-tiered segmentation model, the first tier is the visitor type. This is the traditional visitor segmentation based on persona or relationship. The second tier is a visit or unit-of-work based segmentation that is behavioural and is designed to capture the visit intent. It changes with each new touch. Gary summarises this two-tiered approach as follows: “Describing who somebody is (tier 1) and what they are trying to accomplish (tier 2).”
  2. Capture visit intent – One of the key things that I learned from Gary’s article is the significance of ‘visit intent’ with respect to creating a user-journey model. Visit intent offers an aggregated view of what a visit was about and how successful it was. Both the goal and the success of a visit are important items when analysing a user journey.
  3. 2-tiered segmentation and omni-channel – Gary points out how well his 2-tiered segmentation model lends itself to an omni-channel setup. The idea of 2-tiered segments can be used across any touchpoint, whether it’s online or offline. The intent-based segmentation can be applied relatively easily to calls, branch or store visits and social media posts. The model can also be applied – albeit less easily – to display advertising and email (see Fig. 1 below).
  4. Good starting point for journey analysis – When you look at the sample data structure as outlined in Fig. 1 below, with one data row per user touchpoint visit or unit of work, you can start doing interesting pieces of further analysis. For example, with this abstract data structure you can analyse multi-channel paths or enhance user journey personalisation.
  5. Combine visitor level data with user journey data – It sounds quite complex, but I like Gary’s suggestion to model in the abstract the key customer journeys. This can then be used to create a visitor level data structure in which the individual touchpoints are rolled up. Gary’s example below helps clarify how you can best map different data touchpoints to related stages in the user journey (see Fig. 2 below) .

Main learning point: The main thing that I’m taking away from Gary Angel’s great piece is the two segments to focus on when measuring the user journey: the visitor and their goals. The data structure suggested by Gary lends itself really well to an omni-channel user experience as it combines visitor and user journey data really well.

Fig. 1 – Sample data structure when applying the the 2-tiered segmentation to a user journey data model – Taken from: http://semphonic.blogs.com/semangel/2015/03/a-data-model-for-the-user-journey.html

  • TouchDateTime Start
  • TouchType (Channel)
  • TouchVisitorID
  • TouchVisitorSegmentCodes (Tier 1)
  • TouchVisitSegmentCode (Tier 2)
  • TouchVisitSuccessCode
  • TouchVisitSuccessValue
  • TouchTimeDuration
  • TouchPerson (Agent, Rep, Sales Associate, etc.)
  • TouchSource (Campaign)
  • TouchDetails

Fig. 2 – Example of modelling the acquisition journey for a big screen TV – Taken from: http://semphonic.blogs.com/semangel/2015/03/a-data-model-for-the-user-journey.html

  • Initial research to Category Definition (LED vs. LCD vs. Plasma – Basic Size Parameters)
  • Feature Narrowing (3D, Curved, etc.)
  • Brand Definition (Choosing Brands to Consider)
  • Comparison Shopping (Reviews and Product Detail Comparison)
  • Price Tracking (Searching for Deals)
  • Buying

With an abstract model like this in hand, you can map your touchpoint types to these stages in user journey and capture a user-journey at the visitor level in a data structure that looks something like this:

  • VisitorID
  • Journey Sub-structure
    • Journey Type (Acquisition)
    • Current Stage (Feature Narrowing)
    • Started Journey On (Initial Date)
    • Time in Current Stage (Elapsed)
    • Last Touch Channel in this Stage (Channel Type – e.g. Web)
    • Last Touch Success
    • Last Touch Value
    • Stage History Sub-Structure
      • Stage (e.g. Initial Research) Start
      • Stage Elapsed
      • Stage Success
      • Stage Started In Channel
      • Stage Completed in Channel
      • Channel Usage Sub-Structure
        • Web Channel Used for this Journey Recency
        • Web Channel Used for this Journey Frequency
        • Call Channel Used for this Journey Recency
        • Call Channel Used for this journey Frequency
        • Etc.
    • Stage Value
    • Etc.

This stage mapping structure is a really intuitive representation of a visitor’s journey. It’s powerful for personalisation, targeting and for statistical analysis of journey optimisation. With a structure like this, think how easy it would be to answer these sorts of questions:

  • Which channel does this visitor like to do [Initial Product Research] in?
  • How often do visitors do comparison shopping before brand narrowing?
  • When people have done brand narrowing, can they be re-interested in a brand later?
  • How long does [visitor type x] typically spend price shopping?

Related links for further learning:

  1. http://semphonic.blogs.com/semangel/2015/03/a-data-model-for-the-user-journey.html
  2. http://semphonic.blogs.com/semangel/2015/03/in-memory-data-structures-for-real-time-personalization.html
  3. http://semphonic.blogs.com/semangel/2011/04/semphonics-two-tiered-segmentation-segmentation-for-digital-analytics-done-right.html
  4. http://semphonic.blogs.com/semangel/2015/02/the-visit-is-dead-long-live-the-visit.html
  5. http://semphonic.blogs.com/semangel/2015/02/statistical-etl-and-big-data.html

SQL – Learning about the basic “SELECT” statement

I’m still doing my Stanford online course on relational databases. Today, I learned about the basics of SQL, a special programming language designed for managing data held in a relational database or from stream processing in a .

The teacher of the class, Jennifer Widom, kicked off the class by talking about the difference between a Data Definition Language (‘DDL’) and a Data Manipulation Language (‘DML’):

Data Definition Language (‘DDL’)

  • Create a table in the database
  • Drop a table from the database

Data Manipulation Language (‘DML’)

  • Query the database -> “Select” statement
  • Modify the database -> “Insert”, “Alert” or “Update” statement

Jennifer then told us about the Basic “Select” statement (see Fig. 1 below), explaining that the result of such a statement is to return a relation with a set of data attributes. For example, when you take a simple college admissions database as a starting point where there are 3 relations, each relation having its own set of unique attributes:

  • College ( College Name, State and Enrollment)
  • Student (Student ID, Student Name, GPA and Size High School)
  • Apply (Student ID, College Name, Major and Decisions)

Jennifer then gave us the following examples:

Query involving a single relation

select sID, sName, GPA

from Student

where > 3.6

This query will give you the name and student IDs of those applicants with a GPA higher than 3.6.

Query combining two relations

select sName, Major

from Student, Apply

where Student.sID = Apply.sID

This query will give you data on the names and student IDs for those students applied, filtered by Major.

Jennifer pointed out that SQL is a multi-set model and it therefore allows duplicates. You can eliminate duplicate values by adding the keyword “distinct” to your query. Jennifer also mentioned that SQL is an unordered model which means that you can sort results.

You can include an “order by” clause in your query and add “descending” to order the results of your query:

where Apply.sID = Student.sID

and Apply.cName = College.cName

order by GPA desc, Enrollment;

Main learning point: I found this class about creating a basic “select” statement particularly helpful, as it helped me to get a better understanding of how basic SQL queries are constructed.

Fig. 1 – Elements of the basic “Select” statement in SQL – Taken from: http://www.w3resource.com/sql/sql-syntax.php

 

Select SQL

Learning more about running A/B tests

I love running A/B and multivariate (‘MVT’) tests. These are experiments designed to evaluate different design or copy variants, based on actual performance data. Instead of comparing or deciding on product options based on gut feel, an A/B or multivariate test allows to you to compare alternatives based on objective data and predefined success criteria.

However, running these kinds of tests can be quite tough. These are some of the reasons why:

  • Insufficient traffic – Time and traffic are two important prerequisites if you want to be able to draw some meaningful conclusions from your experiments. However, what do you do if you don’t have a large user base yet or when you traffic starts faltering?
  • Not sure which metric(s) to focus on – One of the things that I’ve learned the hard way is the importance of being clear upfront about the exact goal of the test and ensuring that you’ve selected the relevant metric(s) to focus on.
  • Determine required sample size – Working out the sample size you need to in order to reach a “point of statistical significance” with your test can be tricky. Luckily, most A/B testing tools have an automated function for calculating this.

The other day I came across a great post by Optimizely titled “Stats with Cats: 21 Terms Experimenters Need to Know”. Reading trough this piece really helped me in understanding more about how to best design an experiment and tackle some of the common issues which I outlined above.

These are the main things that I learned from Stats with Cats: 21 Terms Experimenters Need to Know:

  1. Statistical significance – Significance is a statistical term that tells how sure you are that a difference or relationship exists. For example, if you want want to be able to confidently tell whether there’s a difference between version A and B, you need a treshold (e.g. 95%) to describe the level of error you’re comfortable with in a given A/B test. Significant differences can be large or small, depending on your sample size.
  2. Confidence interval – This is a computed range used to describe the certainty of an estimate of some underlying parameter. In the case of A/B testing, these underlying parameters are conversion rates or improvement rates.
  3. Bayesian – This is a statistical method that takes a bottom-up approach to data analytics when calculating statistical significance. It encodes the past knowledge of similar, previous experiments into a prior, which is a statistical device. You can use this prior in combination with current experiment data to make a conclusion on a currently running experiment.
  4. Effect size – The effect size (also known as “improvement” or “lift”) is the amount of difference between the original version (‘control’ version) and a variant. This could be an increase in conversion rate (a positive improvement) or a decrease in conversion (a negative improvement). The effect size is a common input into many sample size calculators. For example, Optimizely’s A/B Test Sample Size Calculator lets you enter an expected conversion rate for your control version.
  5. Error rate – The error rate stands for the chance of finding a conclusive difference between a control version and a variation in an A/B test, or not finding a difference where there is one. This encompasses “type 1” and “type 2” errors. A type 1 error occurs when a conclusive outcome (winner or loser) is declared, and the test is actually inconclusive. This is often referred to as a “false positive”. With type 2 errors, no conclusive result (winner or loser) is declared, failing to discover a conclusive difference between a control and a variation when there was one. This is also referred to as  a “false negative” (see Fig. 1 below).
  6. Hypothesis test – Sometimes called a “t-test”, a hypothesis test is a statistical inference methodology used to determine if an experiment result was likely due to chance alone. Hypothesis tests try to disprove a null hypothesis, i.e. the assumption that two variations are the same. In the context of A/B testing, hypothesis tests will help determine the probability that one variation is better than the other, supposing the variations were actually the same.
  7. Fixed horizon hypothesis test – The key thing with a fixed horizon test is that it’s designed to come to a decision about version A or B at a set moment in time, ideally after reaching the point of statistical significance.
  8. Sequential hypothesis test – A sequential hypothesis test is the opposite of a fixed horizon hypothesis test, as the underlying principle of this test is that the experimenter can make a decision on the test at any point in time.

Main learning point: Even though I’m not a statistician or a data analyst, I found it really helpful to learn more about some of the terms that experimenters need to know about. Especially given some of the challenges with respect to running successful experiments, I believe it’s important to think through things such as a null hypothesis or desired effect size before you design and run your experiment.

Fig. 1 – Possible outcomes of A/B experiments – Taken from: http://blog.optimizely.com/2015/02/04/stats-with-cats-21-terms-experimenters-need-to-know/#type-i Error image

Related links for further learning:

  1. http://blog.optimizely.com/2015/02/04/stats-with-cats-21-terms-experimenters-need-to-know/
  2. http://us6.campaign-archive2.com/?u=ad9057edac5b98ad4892b6a6f&id=bffd19fcf8&e=3c8b6fa69a
  3. http://www.optimizesmart.com/understanding-ab-testing-statistics-to-get-real-lift-in-conversions/
  4. https://www.optimizely.com/resources/multivariate-testing/
  5. http://blog.hubspot.com/marketing/how-to-run-an-ab-test-ht
  6. http://www.wordstream.com/blog/ws/2014/02/26/
  7. http://en.wikipedia.org/wiki/Bayesian_probability