A few months ago, I delivered a talk to an audience of product managers about the importance of having a data informed approach (see Fig. 1 below). As a product person, using data optimally to help make product decisions is critical. It’s something that I try to work on every day. I’m constantly learning about things such as the best combination of quantitative and qualitative, measuring the right things and using data as an integral part of the product development process.

This is a quick summary of the key things that I talked about in my talk in May of this year:

1. Why do we need data? – I’m always keen to stress that as a product manager I don’t have all the answers. Product managers aren’t the holy grail and I believe it would be silly to pretend otherwise. I’m never afraid to say that I don’t know when people ask me which product idea we should go for or how people will use our product. Instead, I’ve learned to use data to draft and test assumptions which can help to inform product decisions (see Fig. 2 below).
2. What can quantitative data tell us? – Quantitative data can really help to get stats on how people actually use a product and measure whether our product improvements or new features have the desired impact. I’ve learned a lot from the book Lean Analytics in which its authors, Alistair Croll and Benjamin Yoskovitz, expand on when and how to best use quantitative data.
3. What can qualitative data tell us? – Qualitative data can be very valuable if you want to find out about the “why” behind quantitative data and to get a better insight into what users think and feel. Also, in cases where you don’t have much quantitative data at your disposal, qualitative data can help you to get some quick input into a product idea or prototype.
4. Data driven approach – There are quite a number of game companies such as Zynga and Wooga that apply a data driven approach to product development. This typically means that a company will pick a single or a set of key metrics to concentrate on (see Fig. 3 below). With a purely data driven approach, its data that determine the faith of a product; based on data outcomes businesses can optimise continuously for the biggest impact on their key metric.
5. Data informed approach – A few years ago, I came across an inspiring talk by Adam Mosseri, Director of Product at Facebook, who introduced the term “data informed” product development. The main rationale for this data informed approach is that, in reality, data is only one of the factors to consider when making product decisions. I make the point that typically, data is most likely to play a role alongside other decision-making factors such as strategic considerations, user experience, intuition, resources, regulation and competition (see Fig. 4 below). This doesn’t diminish the critical nature of data, but it does into take account a reality where other factors need considering when making product decisions.
6. Where and why to use data to help inform product decisions – Also, there are a number of cases where a purely data driven approach falls short. For example, when there’s a strategic decision to be made or when you’re assessing a new product idea (see Fig. 5 below), looking at data in isolation may be insufficient. In my talk, I also gave some examples of how and why I use data at set points of the product lifecycle to help inform decision making (see Fig. 6 below).

Main learning point: I’m looking forward to a book by Rochelle King about data-informed design, hoping to learn more about how and when one can use data to inform product design decisions. I feel like I’ve already learned a lot from people like Adam Mosseri, Alistair Croll and Benjamin Yoskovitz and the way in which they use data as a key factor in making product decisions. Ultimately, I believe that using data continuously – in whichever way or form – is the best way to figure out how to best utilise it.

Fig. 1 – My presentation “Use data to inform product decisions” at ProductTank Hamburg on 16 May 2014 

Fig. 2 – Why do we need data?

• To learn about products and users
• To measure success
• To choose between options
• To understand user interaction
• To decide on our products

Fig. 3 – Key components of a data driven approach

• Focus on the “One Metric That Matters”
• Build hypothesis around key KPI
• A/B or multivariate test continuously
• Optimise your product based on data
• Are we making a noticeable difference?

Fig. 4 – Factors that can play a role in a product decision-making process (this is by no means an exhaustive list!)

• Data
• Strategy
• Intuition
• Competition
• Regulation
• Business
• Brand
• Time
• Technology

Fig. 5 – Reasons a data informed approach might be better suited to your decision-making process

• Data is one of the factors to consider
• Focus on the questions that you want answered
• You can’t replace intuition or creative ideas with data
• Assess impact on relevant areas

Fig. 6 – Examples of how I use to data to inform product decisions

What do we want to do?

Assumptions, hypotheses, assessments and prototypes

How should it work?

User testing, user stories, A/B testing and prototypes

How is it working?

Product retrospectives, tracking and goal-oriented planning

Related links for further learning:

1. http://andrewchen.co/2012/05/29/know-the-difference-between-data-informed-and-versus-data-driven/
2. http://vimeo.com/14999991
3. http://venturebeat.com/2013/12/04/the-4-keys-to-running-a-data-driven-business/
4. http://www.realityisagame.com/archives/390/wooga-follows-zynga-in-metrics-driven-game-design/
5. http://insideintercom.io/the-problem-with-data-driven-decisions/
6. http://www.webdesignerdepot.com/2013/05/the-perils-of-ab-testing/
7. http://www.fastcolabs.com/3033885/ebay-is-running-its-own-sociology-experiments
8. http://www.beyondwebanalytics.com/2013/07/24/episode-67/
9. http://www.slideshare.net/tgwilson/a-process-for-being-data-driven