Data is key. Analytics are key. But what does one need to measure? And why?  Experienced entrepreneurs Alistair Croll and Benjamin Yoskovitz wrote “Lean Analytics”, which came out last month. “Lean Analytics” is a great book for anyone who wishes to learn more about validating an idea, building the right product and measuring growth.

In early 2012, I went to a talk by Eric Ries. It was a great talk, with lots of insights that I have since tried to implement on a day-to-day basis. The only thing missing, however, was a steer on how to measure performance in a ‘lean’ context. I felt that Ries’ suggestions around metrics were very much geared to consumer facing products or business models.

One of the questions I had was around measuring the performance non-consumer facing products such as infrastructural or enterprise services. When I got in touch with Eric Ries to follow up on my questions, the short answer was: “wait for a book by Alistair Croll and Benjamin Yoskovitz, due in March ’13.” I kept patient and it was definitely worth the wait.

This is why:

1. What makes a good metric? – “Lean Analytics” starts off with a basic but very important question: “what makes a good metric?” Croll and Yoskovitz go into four different characteristics of a good metric (see Fig. 1 below) and look at different metric types. For example, I hadn’t yet heard about the difference between ‘leading’ (predictive) and ‘lagging’ (current situation) metrics. This provides an essential starting point for the rest of the book, which delves into which metrics to measure and why. 
2. Be sensible about data – In Lean Analytics, Croll and Yoskovitz do a good job of putting some caveats around the use of data. Firstly, they stress the importance of combining quantitative and qualitative data, urging readers to “get out of the building.” Secondly, they point out the risks of becoming too data-driven. With the help of Monica Rogati, a well-known data scientist at LinkedIn, they outline 10 common data related pitfalls. For example, businesses should be aware of ‘data vomit’; having extensive data dashboards but loosing focus of those key metrics that really matter to your business.
3. The One Metric That Matters – Especially if you’re an early-stage startup, Croll and Yoskovitz stress the value of having the “One Metric That Matters” (OMTM). The reasons they provide to underpin this argument are pretty compelling (see Fig. 2 below). When I work with young startups and their often very driven and passionate founders, I tend to to focus their minds on a single metric or value consideration that is likely to really impact their business.
4. What business are you in? – At first I thought it a bit tedious for Lean Analytics to outline a variety of business models. Surely, most readers should be able to know the difference between an e-commerce model and a media site!? I backtracked very quickly on this thought: not only do Croll and Yoskovitz provide a comprehensive overview of business models to consider, they also offer a great overview of relevant metrics to track per business model. For example, I’m currently working on a free mobile app  and the book offers some useful metrics to consider in this respect (see Fig. 3 below). For each business model the book offers a wealth of concrete suggestions on which metrics to consider and why (using real-life case studies to illustrate).
5. What stage are you at? The kinds of metrics worthwhile tracking are not only dependent on your business model, they are also likely to vary based on the stage your business is at. Lean Analytics breaks down the different stages that most startups are likely to go through (see Fig. 4 below). I believe the value of using these stages as a reference point is twofold. Firstly, each stage requires its own specific metrics. For instance, as a business you’ll most likely concentrate on different metrics at the empathy stage than when you’re at the revenue stage. Secondly, the book suggests you ask yourself at each stage whether you should move on to the next one, using the relevant metrics to validate your decision making.

Learning point: Lean Analytics is a great fit within the “Lean Startup” movement. The book provides a very comprehensive overview of the different metrics worth tracking and – most importantly – offers a clear rationale as to why certain metrics should be tracked for a specific business model at a given stage. The thing I liked most about Lean Analytics is that all of it makes perfect sense. The book provides a well thought through approach to making sense of something that can be complex and challenging at times: data.

Fig. 1 – What makes a good metric? “Lean Analytics”, pp. 9-11

• Comparative – Being able to compare a metric to other time periods, groups of users, or competitors, helps you understand which way things are moving.
• Understandable – If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture.
• A ratio or a rate – Ratios or rates tend to be easier to act on, they are inherently comparative (see the first characteristic above) and they are good for comparing opposing factors.
• Changes the way you behave – What will you do differently based on changes in the metric?

Fig. 2 – Four reasons why you should use the One Metric That Matters – “Lean Analytics”, pp. 58-59

• It answers the most important question you have 
• It forces you to draw a line in the sand
• It focuses the entire company
• It inspires a culture of experimentation

Fig. 3 – Metrics to consider with respect to a free mobile app – “Lean Analytics”, pp. 105-106

• Downloads – How many people have downloaded the application?
• Customer acquisition cost – How much does it cost to get a user and to get a paying customer?
• Launch rate – The percentage of people who download the app, actually launch it and create an account.
• Percent of active users/players – The percentage of users who’ve launched the application and use it on a daily and monthly basis; daily active users (DAU) and monthly active users (MAU).
• Percentage of users who pay – How many of your users pay for anything?
• Time to first purchase – How long does it take after activation for a user to make a purchase?
• Monthly average revenue per user – This can be taken from both purchases and watched ads.
• Churn – How many customers have uninstalled the application or haven’t launched it in a certain time period?
• Customer lifetime value – How much is a user worth from cradle to grave?

Fig. 4 – Common startup stages – “Lean Analytics”, pp. 153-154

1. Empathy – Solving a problem that people care about and are willing to pay for.
2. Stickiness – Create a good product that will solve the problem identified.
3. Virality – Use word-of-mouth to grow your user base.
4. Revenue – Focus more on monetizing your product.
5. Scale – Shift from growing your business to growing your market.

Related links for further learning:

1. http://www.instigatorblog.com
2. http://solveforinteresting.com/
3. http://www.kaushik.net/avinash/
4. http://blog.kissmetrics.com/single-startup-metric/
5. http://www.techopedia.com/2/28887/trends/big-data/big-data-who-to-follow-on-twitter
6. http://www.forbes.com/sites/danwoods/2011/11/27/linkedins-monica-rogati-on-what-is-a-data-scientist/