A few months ago, I delivered a talk to an audience of product managers about the importance of having a data informed approach (see Fig. 1 below). As a product person, using data optimally to help make product decisions is critical. It’s something that I try to work on every day. I’m constantly learning about things such as the best combination of quantitative and qualitative, measuring the right things and using data as an integral part of the product development process.
This is a quick summary of the key things that I talked about in my talk in May of this year:
- Why do we need data? – I’m always keen to stress that as a product manager I don’t have all the answers. Product managers aren’t the holy grail and I believe it would be silly to pretend otherwise. I’m never afraid to say that I don’t know when people ask me which product idea we should go for or how people will use our product. Instead, I’ve learned to use data to draft and test assumptions which can help to inform product decisions (see Fig. 2 below).
- What can quantitative data tell us? – Quantitative data can really help to get stats on how people actually use a product and measure whether our product improvements or new features have the desired impact. I’ve learned a lot from the book Lean Analytics in which its authors, Alistair Croll and Benjamin Yoskovitz, expand on when and how to best use quantitative data.
- What can qualitative data tell us? – Qualitative data can be very valuable if you want to find out about the “why” behind quantitative data and to get a better insight into what users think and feel. Also, in cases where you don’t have much quantitative data at your disposal, qualitative data can help you to get some quick input into a product idea or prototype.
- Data driven approach – There are quite a number of game companies such as Zynga and Wooga that apply a data driven approach to product development. This typically means that a company will pick a single or a set of key metrics to concentrate on (see Fig. 3 below). With a purely data driven approach, its data that determine the faith of a product; based on data outcomes businesses can optimise continuously for the biggest impact on their key metric.
- Data informed approach – A few years ago, I came across an inspiring talk by Adam Mosseri, Director of Product at Facebook, who introduced the term “data informed” product development. The main rationale for this data informed approach is that, in reality, data is only one of the factors to consider when making product decisions. I make the point that typically, data is most likely to play a role alongside other decision-making factors such as strategic considerations, user experience, intuition, resources, regulation and competition (see Fig. 4 below). This doesn’t diminish the critical nature of data, but it does into take account a reality where other factors need considering when making product decisions.
- Where and why to use data to help inform product decisions – Also, there are a number of cases where a purely data driven approach falls short. For example, when there’s a strategic decision to be made or when you’re assessing a new product idea (see Fig. 5 below), looking at data in isolation may be insufficient. In my talk, I also gave some examples of how and why I use data at set points of the product lifecycle to help inform decision making (see Fig. 6 below).
Main learning point: I’m looking forward to a book by Rochelle King about data-informed design, hoping to learn more about how and when one can use data to inform product design decisions. I feel like I’ve already learned a lot from people like Adam Mosseri, Alistair Croll and Benjamin Yoskovitz and the way in which they use data as a key factor in making product decisions. Ultimately, I believe that using data continuously – in whichever way or form – is the best way to figure out how to best utilise it.
Fig. 1 – My presentation “Use data to inform product decisions” at ProductTank Hamburg on 16 May 2014
Fig. 2 – Why do we need data?
- To learn about products and users
- To measure success
- To choose between options
- To understand user interaction
- To decide on our products
Fig. 3 – Key components of a data driven approach
- Focus on the “One Metric That Matters”
- Build hypothesis around key KPI
- A/B or multivariate test continuously
- Optimise your product based on data
- Are we making a noticeable difference?
Fig. 4 – Factors that can play a role in a product decision-making process (this is by no means an exhaustive list!)
Fig. 5 – Reasons a data informed approach might be better suited to your decision-making process
- Data is one of the factors to consider
- Focus on the questions that you want answered
- You can’t replace intuition or creative ideas with data
- Assess impact on relevant areas
Fig. 6 – Examples of how I use to data to inform product decisions
What do we want to do?
Assumptions, hypotheses, assessments and prototypes
How should it work?
User testing, user stories, A/B testing and prototypes
How is it working?
Product retrospectives, tracking and goal-oriented planning
Related links for further learning:
7 responses to “Using data to inform product decisions”
[…] role, the prime ingredient in these decisions is judgment”. I would file this view under the data informed approach to product decision making of which I’m a big […]
[…] brain that allow us to understand other people’s experience. It’s great to use data to inform the design and build of your product, and any decisions you make in the process. The risk with both […]
[…] waiting a while for Rochelle King to write her book using data to inform designs (I wrote about using data to inform product decisions a few years ago). Earlier this year, Rochelle published “Designing with Data: Improving the User Experience […]
[…] P&L accountability for product people (2) – Understandably, this can be a scary prospect and might impact the ways in which we manage products. However, owning the P&L will (1) make product managers fully accountable for product performance (2) provide clarity and accountability for product decisions, (3) help investments in the product and product marketing and (4) steep product management in data, moving to a more data informed approach to product management. […]
[…] P&L accountability for product people (2) — Understandably, this can be a scary prospect and might impact the ways in which we manage products. However, owning the P&L will (1) make product managers fully accountable for product performance (2) provide clarity and accountability for product decisions, (3) help investments in the product and product marketing and (4) steep product management in data, moving to a more data informed approach to product management. […]
[…] If data is available, we’ll present data for each option. […]
[…] Actually, I believe there is an important step that needs to happen before you look at the data: ‘asking the right questions of the data’. What do we want to learn? Why? What are the assumptions that I want to test? How? What does […]