Some considerations regarding data-driven design

Whilst slightly struggling with identifying the most effective measurements for my own product I’m nevertheless learning lots of new things about when (not) to use data in developing products. I’ve been learning more about data-driven product design and some of the key things to consider when using data to inform your product decisions:

  1. Measuring events – What are the key events that we would like to track, and why? This is likely to vary per team or stakeholder. For instance, the number of events that Sales are most interested in is likely to be much smaller than the events that I, as a Product Manager, want to track (see Fig. 1 below). I guess the main thing is here that you pick the appropriate events to measure and set clear goals as to how you would like these events to perform. A good example is Wooga, a German games company, where the product team have a number of KPIs and metrics that they’re looking to deliver on. Each week they’ll pick a KPI e.g. retention and will look at all the activities they can design and measure to increase the chosen KPI. Alistair Croll and Benjamin Yoskovitz have introduced the notion of  the “One Metric That Matters” in this respect, urging businesses to focus on a single metric that will really impact their business.
  2. Retention vs customer engagement – I like to distinguish between customer engagement and customer retention. Often they tend to get lumped together, but in my mind retention is all about if and how often people revisit your site or application. For instance, I find it helpful to do a cohort analysis to compare the number of users that signed up during specific time periods and their revisit rates. These figures should be a good reflection of site performance over time, and the idea being that revisit rates will go up as you continue to improve the site (blogger Andrew Chen has written a great blog post about this). With customer engagement, I tend to be much more interested in metrics such as click-through rates, conversions or discussions around content. Such metrics give a better insight into the extent which users engage with content or a service.
  3. Limitations of A/B and multivariate testing – I’m a big fan of testing multiple versions of a design, whether it’s just to compare two design versions (A/B) or to compare multiple variations of different design elements (multivariate testing). Again, the main challenge here is to ensure you’re testing the right things. You can potentially test a thousand different variables and combinations per web page or application, so I believe it’s critical to start off with the right business questions and to be disciplined about the things that you want to test (see Fig. 2 below).
  4. Data isn’t everything – Whether the data you generate is quantitative or qualitative (or both), I strongly believe that data doesn’t replace product vision or intuition. Data provides a very useful perspective when making a decision, but it shouldn’t be the only factor you’re considering. I know that a lot of my peers disagree with this view, but I’ve identified some constraints over time when it comes to relying on data (see Fig. 3 below). Essentially, I believe that data provide a very valuable lens to look at product performance but data can never be a substitute for ‘going out of the building’ (and talking to customers or competitors) or gut feeling. Data can help in validating intuition or initial assumptions but you’ll need to start somewhere!
  5. Who are you measuring? – I’m learning more about user segmentation and how this can be reflected in the specific things you measure. I found the “See-Think-Do” framework by analytics guru Avinash Kaushik (see Fig. 4 below) very helpful in this respect. It helps to think about specific metrics to measure in relation to specific groups of users. I always find it very helpful to look at analytics within the context of user cohorts, just to get a better perspective.

Main learning point: data can provide a great framework for making business or product decisions. There are numerous professionals and companies out there who make decisions solely based on data. Data are objective and tangible. However, the pitfalls of solely relaying on user data shouldn’t be underestimated in my view.

Firstly, one can easily end up measuring the wrong thing or getting an incomplete picture. Secondly, one can become paralyzed by data, not trusting your product vision and becoming very driven by the users that you have (and not necessarily the users that you want).

If anything, I’ve realised again the importance of establishing a product vision, goals and assumptions first, before you even start contemplating which metrics to measure!

Fig. 1 – Examples of event types that I tend to track

  • Registration landing

  • Registration completion

  • Products entered in based

  • Proceed to checkout

  • Account creation

  • Page view

  • Click-through on search results

  • Documents created

  • Document shared

  • Post created

  • Post updated

  • Post shared

  • Comment created

  • Purchase

  • Visits by a specif group of visitors

Fig. 2 – Things I’ve learned so far about A/B and multivariate testing

What is it that you want to test?

I’ve learned to be very specific about the business questions that I want answers on:

  • I want to improve the conversion rate of the basket checkout process by 5%

  • The aim is to get a 1,000 people to respond to the call to action in the email campaign

  • To see if the number of duration of time spent on a page drops if we introduce page ads

  • To create a shift from “spectators” to “participators”

  • To increase employee productivity from 2% to 5%

  • To convert the current 200 “inactive users” to “active users”, by having them complete at least one activity per month

What does success look like?

This is where I look at Key Performance Indicators and the related metrics to measure and test with:

  • “We know that the new landing page of our site or app is successful if we can reduce the bounce rate by 10%”

  • “We know that the new sign-up process is successful if we manage to increase the sign-up rate by 10% in the first week after launch of the new, simplified sign-up process”

  • “We know that the new “share an update” button is more effective if we see a 5% growth in month-on-month number of status updates shared.”

  • “We know that our user engagement is improving when the user who created a group are still creating groups in March”

Being disciplined about goals and picking the right variables

I’ve learned to try and stick one goal per page to test. Otherwise there’s a risk of things getting messy, making it hard to measure things and – most importantly – to get any meaningful outcomes for your testing in the first place.

Picking the right variables is another key thing. Which elements tend to cause the most friction (e.g. forms, sign-up, page length and process steps)? Which elements are key in achieving your goals? Also, make sure you don’t waste too much time on trivial elements such as text or headlines, because in the context of the key goals that you’re trying to achieve they’re likely to have less of an impact.

Fig. 3 – Some constraints when it comes to relying on data 

  1. Data provides an insight into the ‘what’ but not necessarily into the ‘why’ or ‘how’ – Particularly quantitative data can be great when it comes to monitoring incremental change but is quite limited in providing real customer insight or show which new features create a breakthrough change. I therefore believe that data always needs to be augmented by other perspectives such as user feedback, competitor analysis, etc.
  2. You still need creativity, strategy and intuition (1) – I’ve seen the risk of people thinking that their analytics data were the holy grail, succumbing to ‘analysis paralysis’ or becoming risk adverse, being unable to make decisions without any available data. One could argue that this isn’t necessarily the fault of data but that of the decision making process around it. My point is that data is one – important- source of information to base a product decision on, but it shouldn’t be your sole perspective. For instance, there might be internal business or technical aspects that need to be considered.
  3. You still need creativity, strategy and intuition (2) – Also, customer monitoring can be very reactive, in a sense that you’re following your customers through their experience and optimise accordingly. However, you might want to drive to a specific strategy or have a new product (the relevant metrics to measure might only emerge over time). Ultimately, you will need to be creative or take a leap of faith to get the results that you want.

Fig. 4 – The “See-Think-Do: Measurement Strategy” framework by Avinash Kaushik

Related links for further learning:


Discovering more about social collaboration software

Recently I found myself looking into the world of ‘enterprise collaboration’ tools. The key thing I love about such applications is the amount of transparency they offer. Suddenly, discussions, thoughts, suggestions and documents become a lot more visible and accessible.

In a previous role I had worked a lot with tools like Jive and Basecamp and I’ve recently been ‘collaborating’ a lot through Asana and Yammer. It made me realise that even though a lot of these tools set out to provide a similar value or proposition, there are nevertheless some differences worth looking into:

  1. Online collaboration vs online project management (1) – People can collaborate around ideas or specific projects, or both. Yammer is great as a tool to collaborate around ideas whereas Basecamp and Asana are more geared towards project management. As my ex-colleague Daniel Siddle – who specialises in this area – put it: “real-time collaboration is a hard one to get right since the concrete end goal can be much harder to define and less tangible compared to using online project management software.” With project management software the tangible outcome is that you can deliver a project faster but with social collaboration software things can be a lot less tangible.
  2. Online collaboration vs online project management (2) – What I like most about using tools such as Yammer, Jive, Chatter (Salesforce) and Confluence is that they enable full transparency, keeping all relevant communications in a single place. When working on specific projects, tools such as Podio (see Fig. 2 below) and Basecamp (see Fig. 3 below) can provide visibility on project progress and on who’s doing what. One thing I learned from having another play with some of these tools is that most online collaboration tools also seem to have at least some project management functionality. Good examples in this respect are Yammer, Tibbr and IBM Connections. Employees can have lengthy discussions on these platforms but are able to switch into a more project management related part of the system if required. In contrast, some of the online project management tools that I’ve looked at seem less geared towards open collaboration.
  3. Some tools in the online collaboration space and what to look out for – Tools that come to mind are: Chatter, SocialtextIgloo, Jive, Yammer, Confluence, MangoApps and daPulse (see Fig. 1 below). As with any digital application, key things to look out for are (1) ease of use and clean interface design and (2) management of information. With some of the tools that I mentioned above there’s a risk of information overload, with the application becoming one long activity stream. Also, I’ve learned from implementing some of these tools with clients that the more intuitive it is to share and comment on ideas, the higher the uptake of these tools. I like tools such as Yammer and Jive because they are so intuitive and easy to use.
  4. Some tools in the online project management space and what to look out for – When managing projects of any scale and with a number of different people involved, Gantt charts or emails are no longer sufficient in my view. Tools like Asana, Basecamp, Podio, Trello and SocialCast provide private workspaces dedicated to specific projects and make it easier to keep track of project progress and outstanding tasks. Whereas a tool like Yammer is continually strengthening the project management aspect of its application (see Fig. 4 below), I find that Asana, Podio and Basecamp (see Fig. 2/3 below) can really help in assigning tasks as well as understanding the status of a project and its individual milestones. Another aspect to look out for is the secure sharing of documents. Most of the applications I mentioned above do have that capability, but there also platforms out there such as Dropbox and Box that do just that: securing storing and sharing of documents.

Main learning point: having used social and project management tools for a while now, it’s interesting to see an overlap in functionality and in proposition arising between the different tools. Like with most products the main challenge to the user is to be clear what they want get out of a specific tool and to establish whether it can deliver on that expectation. For instance, if one’s end goal is to deliver projects faster, some of the open collaboration solutions might not be appropriate. In contrast, if one likes to collaborate around ideas then the more traditional project management software might not be the way to go.

Fig. 1 – An introduction to daPulse by daPulse

Fig. 2 – How Podio can be used for Project Management by Podio

Fig. 3 – A review of the Basecamp project management functionality by Joel Milne

Fig. 4 – An overview of new Yammer features by Yammer

Related links for further learning: