How to do effective user interviews?

A few years ago, I did a really useful online class on user interviews, which was instructed by Julie Blitzer. Julie is an experienced UX designer/strategist who shared some great learnings and tips on how to best do qualitative user interviews. This is the main thing that I learned from Julie’s session:

listen-with-social-media-first-2-800x288

Yes, you guessed it right: LISTENING is everything when it comes to interacting with (target) users. Julie talked about the importance of listening and not asking leading questions, and I’ve been able to put both lessons in practice since.

However, having just started a new job as a product manager at Beamly, I thought it would be good to refresh my knowledge of user interviews and to look for new, practical tips that I could benefit from:

  1. What do you want to get out of the user interview?  A key part of the effectiveness of user interviews is in the preparation. I believe that there are two critical aspects to the interview prep. Firstly, defining the more strategic outcomes that you would like to get of the user interviews. What is that it that you want to learn from the user? Which user or business assumptions do you wish to validate? You can use these kinds of questions to formulate the goals of your user interviews. In his book “Inspired: How To Create Products Customers Love”, product management expert Marty Cagan outlines how one can utilise user interviews (or other product discovery techniques) to help answer some important product questions (see Fig. 1 below). Secondly, preparing all the different practicalities involved in user interviews is just as crucial an aspect of user interviews as the way in which you run these sessions. I’ve listed some practical things in Fig. 2 below which you might want to consider.
  2. Avoid leading questions – I’ve learned to refrain from asking users questions such as “You probably won’t get all the information that you need out of this landing page?” or “This is probably not the best way to navigate this app?”. Questions like these almost constitute a cardinal sin when it comes to generating true customer insights since you’re leading the user in a certain direction, giving him/her the idea that there’s a specific response that you’re looking for.
  3. Keep it open ended – I’ve learned that it’s better to leave the questions fairly open-ended and to avoid leading questions or ones that can be answered with a simple ‘yes’ or ‘no’. Better questions are in my view the ones that encourage users to explain their thinking or experiences. For example: “What about this landing page do you like? “What don’t you like?” “How do you typically find out about new music?” As part of the introduction to the session, I always explain to users that this isn’t an exam and that aren’t any right or wrong answers.
  4. Avoid “Would you like this Ferrari?” type questions – My experience is that when you test a prototype or a new product with users they are almost all likely to answer ‘yes’ when you ask them whether would buy or use this product. I feel that is therefore a redundant question, answers to which are unlikely to tell you very much. No one can (accurately) predict future behaviour and there are so many factors that will impact actual product usage, that asking ‘would’ questions will provide you with fairly meaningless insights in return. I have heard a good number of war stories where product people made the mistake of thinking that their product would be a great success, purely based on people saying during interviews that they would buy it.
  5. Combine user interviews with live prototypes – Ideally, I prefer doing user interviews in conjunction with people actually using your product, even if it’s still in its most ‘minimal’ state. Whether a product is being used in Beta or has been launched fully, tracking usage data will only help you to learn so much (I wrote about the constraints of pure data-driven design last year). User interviews provide a great way to find out more about the ‘why’ behind the analytics data that you can track.
  6. Listen, reflect and probe – Listen, be silent … these things are easier said than done when you’re a passionate or opinionated product person but in my view oh so important when doing user interviews. When I explain the purpose and format of the interview to users, I tend to ‘warn’ them that I might not always respond (immediately) to their questions and that I will at times ask them to ‘think out loud’. This might sound a bit forced and unnatural, but understanding users’ thought processes, considerations or steps that they go through can be really insightful. UX people like Grace Ng, Adrian Howard and Michael Hawley have written some good, practical pointers on how to best paraphrase users and to ask ‘why, what, when or how’ questions to probe further.

Main learning point: It was good to refresh my understanding of some the tips around how to best do user interviews. However, I do feel that conducting these user sessions boils down to one fundamental thing: active listening. A lot of the practical tips and techniques that one can apply are great, but I do believe that the ultimate skill required is to truly listen to what the user is saying (and to what the user isn’t saying).

Fig. 1 – Key product questions that user interviews (or other product discovery techniques) can help to address (taken from: Marty Cagan – “Inspired: How To Create Products Customers Love”, Chapter 16)

  • Do you understand who your users really are?
  • How are users using your product?
  • Can users figure out how to use your product? Where do they stumble?
  • Why do users use your product?
  • What do users like about your product?
  • What do users want added to or changed in your product?

Fig. 2 – Some practical things to consider when preparing and running user interviews:

When approaching users to participate:

  • Think of the users you’d like to engage with – Whether you use a recruitment agency like Indiefield, a site like ethnio or if you recruit people yourself, it’s important to have a clear idea of the target persona(s) that you’d like to interact with.
  • Include the purpose, location and duration of the interview session – Inform the user upfront on the nature of the session, its expected duration and provide the details of the place where the interview will take place.
  • If you’re offering users an incentive, outline details  I typically include details on the relevant incentive in the initial email (e.g. an Amazon voucher).
  • Provide users with a range of dates and time slots to chose from – There are tools that automate the process of scheduling slots but you can always use a simple spreadsheet to keep track of available dates and time slots.

Preparing the questions and the room:

  • Create a high level interview outline – This is not a detailed script! This outline should provide you with a guide and reference point to use throughout the sessions. For example, an outline can contain any of the following elements: (1) interview objectives (see point 1. above), (2) interview topics, themes or scenarios that you want to cover and (3) some initial questions or tasks related to each part of the interview (again, these don’t have to be super detailed, but should help to get the conversation going.
  • Create a user consent form – The main purpose of this document is to ensure that the user is happy with you recording the session and using the data from the interview for internal purposes only.
  • Get a tape recorder and/or camera – Use your phone or tape recorder to record the session. Not only will this help in you listening back to the interview in full, it also means that you don’t spend the entire session taking notes. Instead, you can instead focus on listening.
  • Get a comfortable chair and a drink – I know it sounds obvious, but do make sure the user feels comfortable in the room or location where you’re conducting the interview.
  • Don’t create a tribunal setting – Especially if you have two people conducting the interview (i.e. one person asking questions and one taking notes) I think it’s important to avoid create a setting where the users feels that he/she’s being interviewed for a job or grilled as part of an exam.

Facilitating the session:

  • Do at least one trial run – I’ve learned – the hard way – of the importance of doing at least one trial run of your user interview of your usability test. This will give you the opportunity to iron out any kinks or detect any questions/tasks that don’t make sense to the user.
  • Prepare one other person – Ideally, I like facilitating these sessions with one other person. This person can ask the user questions, take notes or simply observe. I find this a great way to involve other people within the organisation, e.g. engineers or marketing people.
  • Introduce the interview session – Again, this is a bit of a no-brainer but it’s nevertheless very important to explain to the user about the nature of the session, the things that you’re hoping to learn and to ensure that the user feels comfortable asking you questions or seeking clarification.

Related links for further learning:

  1. http://uxdesign.smashingmagazine.com/2014/04/10/validate-product-ideas-with-quick-simple-experiments/
  2. https://www.quora.com/Customer-Development/What-are-your-favorite-questions-to-ask-customers-when-validating-a-product-or-feature
  3. http://uxceo.com/post/80877539095/quick-tips-for-effective-customer-interviews
  4. http://svpg.com/product-discovery-with-live-data-prototypes/
  5. http://theuxreview.co.uk/user-interviews-the-beginners-guide/
  6. http://viget.com/inspire/user-interview-tips-for-non-ux-designers
  7. http://www.slideshare.net/adrianh/leanux-effectivecustomerinterviewingga20130606
  8. http://www.slideshare.net/edanzico/user-interview-techniques
  9. https://medium.com/hubspot-product/4-customer-interview-questions-you-should-never-ask-a5596586313b

Which principles do underpin game design?

Gamification is more than just creating some mechanisms around rewards and motivation.  In his online lectures on gamification, Kevin Werbach – an Associate Professor of Legal Studies and Business Ethics at Wharton School – explained about some of the design rules which underpin gamification and game design:

  1. The Player Journey – Like with most journeys, games tend to have a beginning, a middle and an end. Kevin explained that the journey starts with “onboarding”; finding a way to get the player in the game. The next step is “scaffolding”, which involves helping the game player to play the game. Finally, “pathways to mastery” are all about the player perfecting his/her grasp of the game. A good example is the game “Plants vs Zombies” which offers its players feedback, limited options and makes it impossible for players to fail. These aspects of the player journey are there to make it as easy as possible for players to get into the game without having to refer to a manual.
  2. Balance in the game – In his lectures, Kevin stressed the importance of ensuring that the game is constantly in balance, not making the game too hard nor too easy. A simple example is “Monopoly”. This well known board game is all about about keeping a virtual economy in balance.
  3. Create an experience – Kevin stressed that gamification and game design are more than just throwing a few game components together. Instead, it’s about a creating an experience. He mentioned Turntable.fm as an example of a game where it wasn’t just about listening to music, but an interactive experience which was designed to recreate a ‘club feel’.
  4. Tapping the emotions of the player – Question: What makes games engaging? Answer: fun. This is one of the reasons why gamification often tends to be utilised in a work context, making work or behaviour change fun. It was interesting to hear about Marc LeBlanc’s “8 kinds of fun” (see Fig. 1 below) and Nicole Lazzaro’s “Four Keys to More Emotion without Story” (see Fig. 2 below). As Kevin talked through a large number of things which can be perceived as fun, he emphasised that these different forms of fun don’t have have to be mutually exclusive. For example, games can be designed to combine forms of ‘hard’ and ‘easy’ fun.

Main learning point: It’s so easy to underplay the value of games and the design thinking that goes into them. It can be just as easy to dismiss games as meaningless fun, but I guess the main thing that I learned from Kevin Werbach’s online lectures on gamification principles is that all aspects of the player journey should have some form of meaning to the player. Whether it’s about tapping into player emotions or creating an interactive experience, all these principles need to be taken into account when thinking about a game and its outcomes.

Fig. 1 – Marc LeBlanc’s “8 kinds of fun” (taken from: http://8kindsoffun.com/)

  1. Sensation
  2. Fantasy
  3. Narrative
  4. Challenge
  5. Fellowship
  6. Discovery
  7. Expression
  8. Submission

Fig. 2 – Nicole Lazzaro’s “Four Keys to More Emotion Without Story” (taken from: http://xeodesign.com/xeodesign_whyweplaygames.pdf)

Hard Fun (‘Emotions from Meaningful Challenges, Strategies and Puzzles):

  • Playing to see how good I really am
  • Playing to beat the game
  • Having multiple objectives
  • Requiring strategy rather than luck

Easy Fun (‘Grab Attention with Ambiguity, Incompleteness, and Detail’):

  • Exploring new worlds with intriguing people
  • Excitement and adventure
  • Wanting to figure it out
  • Seeing what happens in the story, even if I have to use a walk through
  • Feeling like me and my character are one
  • Liking the sound of cards shuffling
  • Growing dragons

Altered States (‘Generate Emotion with Perception, Thought, Behaviour, and Other People’):

  • Clearing my mind by clearing a level
  • Feeling better about myself
  • Avoiding boredom
  • Being better at something that matters

The People Factor (‘Create Opportunities for Player Competition, Cooperation, Performance, and Spectacle’):

  • It’s the people that are addictive not the game
  • I want an excuse to invite my friends over
  • I don’t like playing games, but it’s a fun way to spend time with my friends
  • I don’t play, but it’s fun to watch

Related links for further learning:

  1. http://www.vanderbilt.edu/econ/faculty/Vrooman/vrooman-rio-sports-special.pdf
  2. http://www.yukaichou.com/gamification-study/4-experience-phases-gamification-3-scaffolding-phase/
  3. http://gamification21.wordpress.com/learning-content-2/7-tapping-the-emotions-fun/
  4. http://xeodesign.com/xeodesign_whyweplaygames.pdf
  5. http://www.thefuntheory.com/
  6. http://www.redkeybluekey.com/2011/09/8-principles-of-good-game-design.html

My learnings from Lean Day London ’14

Two weeks ago, I attended the London edition of Lean Day, a 2-day event dedicated to “changing the way products are built”. Put together by Jeff Gothelf and his colleagues at Neo, it was first time this event was being held in London. The main theme of Lean Day was around improving the way in which we create or iterate products.

As always with inspiring events, you come home with your head buzzing with new ideas and thoughts. I took some time to reflect and forced myself to narrow things down to a maximum of 5 things that I’d like to implement in my day-job in some form over the next 1-3 months:

  1. Focus on ‘jobs’ and determine where a product starts and stops – The talk at Lean Day which stood out for me was the one by Des Traynor, Co-Founder of successful Irish startup Intercom. It was interesting to hear Des talk about the “irony of product management”: “because you’re trying to make everything better, you make everything worse.” He therefore urged us to focus on a limited set of jobs instead (see Fig. 1 below). This approach really helps to narrow down the product scope and the specific user problem that you’re trying to solve. In a similar vein, Des also talked about the importance of determining where a product starts and stops (see Fig. 2 below).
  2. Estimating impact – In his talk at Lean Day, Agile coach Gus Power talked about impact estimation. Taking the work of Gojko Adzic on Impact Mapping as a starting point, Gus talked about measuring the impact of different design ideas on the desired – business or user – outcome. Impact mapping and estimation can really help in thinking through the potential value of a certain idea or feature as well as the underlying assumptions. Similar to Des Traynor, Gus emphasised that it’s not about the rate at which you build new features, but much more about their impact and their subsequent iterations.
  3. “Weighted Shortest Job First” –  It was interesting to hear Robin Pembrooke, Head of Product at BBC News Online, talk about the “Weighted Shortest Job First” (‘WSJF’) as a way to prioritise software development. Robin spoke about inheriting complex legacy products, for which it’s quite useful to use the WSJF method to prioritise. WSJF was first introduced by Donald Reinertsen in his book “The Principles of Product Development Flow” (2009) and is a useful way to first prioritise those features that take a relatively little amount of time but which do contribute to user/business value and to ‘time criticality’ (cost of delay).
  4. Customer exposure hours  Leisa Reichelt, who’s Head of User Research at the Government Digital Service (‘GDS’), spoke about how they do ‘customer exposure hours’ at the GDS. About every 6 weeks, they’ll dedicate 2 hours to meeting with GDS customers, watching real users interact with their site or with new designs. Lisa made a good point in her talk about the collaborative nature of user research; stressing that “user research is a team sport.” For example, with the regular customer exposure hours at the GDS, the whole digital team tends to get involved (including senior stakeholders).

Main learning point: Going back over my notes from the 2-day “Lean Day London” conference, the two words that best capture the majority of talks are “collaboration” and “iteration”. I loved the fact that most speakers were passionate about learning and doing product experiments. However, the speakers made it clear that even for such an iterative or “lean” approach, an understanding of your (target) users and success factors are important prerequisites. Over the next 1-3 months I’m therefore looking to focus more on user understanding and how to best determine what success looks like, both from a users and a business point of view. All in all, I felt that Lean Day London is one of the best conference that I’ve been to for quite a while!

Related links for further learning:

  1. http://scaledagileframework.com/wsjf/
  2. http://scalingsoftwareagilityblog.com/prioritizing-features/
  3. http://www.uie.com/articles/user_exposure_hours/
  4. http://sjhcockrell.com/writing/2013/john-cleveley-responsive-news/
  5. http://www.fastcompany.com/52717/change-or-die
  6. http://rsbatechnology.co.uk/blog:8
  7. https://signalvnoise.com/posts/3723-the-category-moat
  8. http://bokardo.com/archives/how-to-embrace-and-extend-product-categories/
  9. http://www.agileproductdesign.com/

Fig. 1 – Think about “jobs” when developing a product (based on my notes from a talk by Des Traynor at Lean Day London 2014, examples are my own)

The Job = The Problem + The Situational Context + Success Criteria

The Problem:

“What outcome are your users trying to achieve?”

I’m struggling to find new music content

I find it hard to keep up to date with what’s happening within my favourite music genre 

I don’t know what to listen to anymore or which concerts to go to

The Situational Context:

This problem typically occurs when I’m at work, wanting to ‘get into the zone’ and am looking for new music to listen to

I don’t want to spend hours creating a new playlist

When my friends ask me to come to a show with them and I’m not familiar with the artist they’re talking about

Success Criteria:

What matters to the actual users?

It needs to be fast

Easy to use

Easy to customise

Fig. 2 – Determining where a product starts and stops (based on my notes from a talk by Des Traynor at Lean Day London 2014)

Start at the first step in the workflow where you can add unique value, and work out how to make the transition from the previous step seamlessly.

Stop when:

There’s a well defined market leader (and you don’t want to compete).

The job(s) is completed in various different ways.

Hardly anyone is using your product or feature.

It’s something you van’t delivery any value on.

Fig. 2 – Weighted Shortest Job First  (taken from: http://scalingsoftwareagilityblog.com/prioritizing-features/)

WSJF

 

 

 

 

Ben Essen and The Quantified Self

The other day I heard a few people use the term “quantified self”. Through Wikipedia I learned that the quantified self stands for “a movement to incorporate technology into data acquisition on aspects of a person’s daily life in terms of inputs, states, and performance.” In other words, this is all about quantifying peoples’ lives and behaviours, thus being able to learn more about people and their different activities and needs.

Ben Essen, a London-based Creative Strategy Director, recently talked about the quantified self at this year’s SXSW in Austin. His talk was titled “Know Thyself. Self Actualization By Numbers” and these are the main things that I learnt from Ben’s presentation:

  1. Essen’s Hierarchy of Quantified Self – Similar to Maslow’s Hierarchy of Needs, Ben Essen has come up with his own “Essen’s Hierarchy of Quantified Self” (see Fig. 1 below). Ben’s hierarchy starts with “goal-progress” (e.g setting daily goals with apps such as Fitbit and Wello) and ends with the “Quantified Society” where everything is informed by our personal data. I like how Ben’s ‘pyramid’ moves from “insight” to “enhancement”, thus highlighting the changing role of personal data as one moves up along the hierarchy.
  2. Context-driven measurementMelon is a great example with regard to quantifying personal data within a specific context. For instance, with Melon you can track how focused you are when you’re working on your laptop compared to when you’re meditating (see Fig. 2 below). Ben refers to this as “lifestyle context”, which implies that your personal data are likely to vary dependent on your mood or the activity that you are doing. Another good example is Nest which home products are designed to learn from user behaviour. I’ve written a few posts on wearable devices and wearable trends to look out for.
  3. “The Human API” – Ben ultimately envisages a ‘Human API’ which encapsulates all your personal data, irrespective of the underlying data source (e.g. email, browse history, search, etc. – see Fig. 3 below). I’ve been trying to visualise an API of all my personal data (e.g. “went to a Danny Brown gig last month, purchased “The Mindful Leadership” on his Kindle and checked in with his Oyster in north London this morning”) and how a brand or other 3rd party would tap into this data set. This concept provides both opportunities (e.g. fully personalised experiences) as well as risks (e.g what happens if my Human API falls into the wrong hands?).
  4. Connecting data sets and devices – I strongly believe that the next frontier in digital development is the connection of different devices and the connection of a user’s various data sets. The possibilities are endless, but I reckon it will take a while to properly connect personal devices and data, thus creating a ‘personal platform API’ similar to the “Human API”, as mentioned by Ben in his talk at SXSW.
  5. Data shouldn’t replace our intuition – I personally prefer using the term “data informed” over the more common “data driven” since I feel that there are some strong limitations to a purely data-driven approach (I’ve blogged about these constraints previously). In his talk, Ben stressed the importance of understanding and interpreting personal data and using data as a source for decision making. However, Ben was keen to stress that “self-tracking must feed our intuition. Not replace it.”

Main learning point: Ben Essen has got a lot of interesting and thought-provoking insights around the topic of the “quantified self”. We are moving steadily in the direction of a society where a lot our behaviours, mood states and activities can or have already been quantified. The idea of a quantified self and a “Human API” will in my opinion truly materialise once we all get smarter about how we connect different devices and data sets. In the meantime, I suggest looking into some of Ben’s observations and reservations around self-tracking and have a think about about how we can move up “Essen’s Hierarchy of Quantified Self”.

Fig. 1 – Essen’s Hierarchy of Quantified Self (taken from: http://www.slideshare.net/benessen/knowthyself-sxsw-benessen) Essen's hierarchy if qs Fig. 2 – Melon Kickstarter video (taken from: https://www.kickstarter.com/projects/806146824/melon-a-headband-and-mobile-app-to-measure-your-fo)

Fig. 3 – The Human Api by Ben Essen (taken from: http://www.slideshare.net/benessen/knowthyself-sxsw-benessen) The Human API

Related links for further learning:

  1. http://www.slideshare.net/benessen/knowthyself-sxsw-benessen
  2. http://www.wired.com/insights/2013/11/the-human-api-dont-forget-the-people-for-the-program/
  3. http://gigaom.com/2013/06/17/startup-human-api-wants-to-bring-quantified-self-data-into-the-mainstream/
  4. http://blog.programmableweb.com/2013/07/02/the-quantified-self-growing-interest-in-apis-to-manage-personal-data/
  5. http://quantifiedself.com/2011/09/robin-barooah-i-am-broken-or-i-can-learn/
  6. http://www.computerweekly.com/news/2240212466/Google-bets-on-internet-of-things-with-Nest-acquistion
  7. http://quantifiedself.com/2012/12/get-your-mood-on-part-1/
  8. http://www.digitalmcgyver.com/personal/gadgets/moves-app-great-for-life-logging-worthless-for-quantified-self/
  9. http://www.slideshare.net/mark_pdx/mark-leavitt-using-data-to-hack-my-habits-and-whip-up-my-willpower-show-and-tell-for-qs-europe-may-2013