Heuristics – Finding usability problems in designs

As part of my online Human Computer Interaction (‘HCI’) course by Scott Klemmer, I learned a lot about “heuristic evaluation”. Even though I had heard the term ‘heuristics’ used before, I had no clue what this entailed. In his online lecture, Scott explained that heuristic evaluation is a method one can use to review user interface (‘UI’) designs. He was quick to stress that there are multiple ways to evaluate designs and that heuristic evaluation is just one of them (I’ve outlined some of the different review methods in Fig. 1 below).

In essence, heuristic evaluation means that you get experts or peers to critique your UI design. Typically, the evaluators use a set of principles (“heuristics”) to assess the design. In 1995, UX expert Jakob Nielsen introduced his “10 Heuristics for User Interface Design” as outlined in Fig. 2 below. These criteria can serve as a good guidance for one’s design evaluation process. However, none of these heuristics are set in stone; the actual criteria that one ends up using are very likely to depend on one’s specific design goals.

Let’s zoom in a on the following, more ‘practical’ aspects of heuristic evaluation:

  1. When to get design critique? Scott suggested a number of stages at which it can be valuable to get design critique. I guess the main thing here is to have a clear design goal and an understanding of what it is that you would like to get out of peer design reviews. In Fig. 3 below I’ve outlined some stages at which you can ask peers or experts to do a design review.
  2. The different phases of heuristic evaluation – In Fig. 4 below I have included some common phases of the heuristic evaluation process. Ideally, you have 3-5 evaluators reviewing your designs. It’s preferable to have multiple evaluators as different people are likely to find different problems. You can then have them compare their findings afterwards. You don’t need a fully working UI to do get peer input, you can just as well have your evaluator review some paper sketches!
  3. How to rate designs? It was interesting to learn more about how to review designs. Scott suggested to use a severity rating which could combine the following factors: frequency, impact and persistence. Scoring a problem will help in establishing the importance of resolving a certain usability problem and allocating resources to fix it. I’ve added a range for such scores – as suggested by Scott – in Fig. 5 below.
  4. Heuristic evaluation vs user testing – Scott emphasised that heuristic evaluation and usability testing with real users aren’t mutually exclusive. He talked through some of the pros and cons of both feedback methods and stressed the value of alternating both approaches. By mixing up methods you’re likely to find different problems and you’re less likely to waste user participants’ time.

Main learning point: heuristic evaluation is something that I hadn’t applied previously, but it was really great to learn more about this review technique and how to best apply it. By having both ‘experts’ and real users testing your user interface designs you’re more likely to get as rounded product feedback as possible.

Fig. 1 – Multiple ways to evaluate UI designs 

  • Real-time feedback from users (which I wrote about earlier – insert link to rapid prototyping post)
  • Use formal models to predict user behaviour
  • Heuristic design evaluation – having peers review and critique your UI
  • Automated user acceptance tests (thinks tools like Cucumber and Specflow)

Fig. 2 – Jakob Nielsen’s 10 Heuristics for User Interface Design (taken from: http://www.nngroup.com/articles/ten-usability-heuristics/)

  1. Visibility of system status – The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
  2. Match between system and the real world – The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
  3. User control and freedom – Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
  4. Consistency and standards – Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
  5. Error prevention – Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
  6. Recognition rather than recall – Minimise the user’s memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
  7. Flexibility and efficiency of use – Accelerators — unseen by the novice user — may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
  8. Aesthetic and minimalist design – Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
  9. Help users recognise, diagnose, and recover from errors – Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
  10. Help and documentation – Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

Fig. 3 – Stages at which you can ask peers or experts for a design review (by Scott Klemmer)

  • Before user testing -> Objective: to identify and resolve minor issues before inviting real users for testing.
  • Before redesigning -> Objective: to get a better understanding of what works and what needs changing.
  • Articulate problems -> Objective: to gather concrete evidence for usability problems (or to make a case for a redesign).
  • Before release to LIVE -> Objective: to smooth out any remaining rough edges prior to going live with a new product or feature.

Fig. 4 – Phases of heuristic evaluation (by Scott Klemmer)

  1. Pre-evaluation training: provide the evaluators with the required domain knowledge and relevant scenarios to review against
  2. Heuristic evaluation: individuals evaluate and aggregate results
  3. Apply severity ratings: at this stage, you determine how severe each usability problem is (and prioritise accordingly). This can be done individually first and then discussed as a group.
  4. Debriefing: review findings of the heuristic evaluation with design/development team.

Fig. 5 – Suggested range for severity scores (by Scott Klemmer

0 = we don’t think this is a usability problem

1 = this is a cosmetic problem

2 = we feel this is a minor usability problem

3 = we feel this is a major usability problem, important to fix

4 = this is usability catastrophe, imperative to fix

Related links for further learning:

  1. http://www.nngroup.com/articles/ten-usability-heuristics/
  2. http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
  3. http://www.nngroup.com/articles/usability-problems-found-by-heuristic-evaluation/
  4. http://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/

Book review: “Developing Products in Half the Time”

Partly due to the advocates of the Lean Startup movement, these days lots of people in the product space talk about rapid product development and looking at ways to get their products to market faster. Actually, one could argue that the roots of this idea are much older and the book “Developing Products in Half the Time” written by Donald Reinertsen and Preston Smith serves as proof of that. This first edition of this book was published in 1991 and still holds true for a lot of the change processes that many companies are going through.

Reading “Developing Products in Half the Time” provided me with some great insights and practical pointers in relation to both making product tradeoffs and managing the product development process:

  1. You still need rational decision making (even with rapid product development) – Early on in the book the authors stress the importance of putting the subject of rapid product development on a rational business basis. I can imagine it might be tempting to let the word “rapid” take over the product decision making process, but Reinertsen and Smith make a convincing case for having clear business questions (and drivers) form the foundations of the product development process.
  2. Doing the product right (vs doing the right product) – “Developing Products in Half the Time” does not tell its readers how to build the ‘right’ product, it leaves that to (product) managers to ensure there’s a clear product strategy in place. In contrast, the book concentrates on “doing the product right” and highlights 4 key objectives (and 6 trade-offs) to consider in this respect (see Fig. 1 below).
  3. The basic modelling process – For me, the greatest value in reading “Developing Product in Half the Time” was the chapter on creating a baseline model to help calculate project product profits over time. I wouldn’t necessarily treat this tool as something that you do as a one-off and then expect (or hope) the product to perform accordingly. Instead, the variables that go into the baseline model can be easily adapted to a range of different situations (e.g. a development expense overrun, a unit cost overrun or a schedule delay- see Fig. 2 below for an example). Similar to when creating a business case, what I liked about creating a baseline model and its subsequent variations are the financial and performance related assumptions which underpin the model.
  4. Decision rules – As Reinertsen and Smith point out: “the true value of this analysis comes from converting the results into a set of decision rules for making tactical trade-offs on the project.” Once you have used the baseline model and its variations to work out the profit impact, you can convert this profit impact analysis into decision rules like “I’m only going to allow a 1% cost overrun on this product development project” or “1 month is the maximum delay that we can have for the introduction of this product.”
  5. Use the baseline model and the opportunity assessment – Whilst reading the book it dawned on me how I could use the baseline model and Marty Cagan’s Opportunity Assessment in conjunction. This way I could size a product opportunity in both a strategic and a more financial kind of way. It was interesting to read about the “market clock” in the book. The market clock starts ticking when a customer opportunity appears and only stops when the customer need is fulfilled. “The market clock is unforgiving”, Reinertsen and Smith explain. This means that project costs will continue to accumulate even when no one is working on a project. Hence the importance of considering the “cost of delay”, irrespective of whether the delay occurs at the start or end of a project. In short, the key point behind “Developing Products in Half the Time” is all about how quickly one can respond to market opportunities.
  6. Pitfalls of incremental innovation – I found it refreshing to read a honest assessment of potential risks and pitfalls related to incremental innovation (see Fig. 3 below). As much as I’m a fan of business agility, I think it’s important to consider some of the risks related to iterative development. In “Developing Products in Half the Time” the authors spend a chapter outlining some approaches to identifying, assessing and managing risk – technical or otherwise – which I found very helpful.
  7. Periodic prototyping – Like authors such as Steve Blank, Eric Ries and Ash Maurya (who all came after “Developing Products in Half the Time”), Reinertsen and Smith are all about “periodic prototyping”, creating prototypes at regular, pre-established intervals. I guess that is one of the critical success factors to lean product development; doing certain things constantly in an iterative fashion. I strongly believe that the whole point behind this approach is continuous learning and improvement!

Main learning point: if you’re interested in product development, I recommend you read “Developing Products in Half the Time”. You might feel that not all of the book’s points apply to your product or organisation, but it should nevertheless give you some principles and practical tools to consider or to apply. For me, the main things I took away from reading “Developing Products in Half the Time” were the creation of a baseline model and considering the cost of delay, which will definitely come in handy next time I need to make a business case or decide on a trade-off.

Fig. 1 – Four key product development objectives and six trade-offs (from “Developing Products in Half the Time” by Donald Reinertsen and Preston Smith, p. 23)

  • Market introduction date: the date when the product is first available for sale
  • Product unit recurring cost: this is a combination of manufactured cost of the product and any variable cost that appear below the gross margin line
  • Product performance: the product revenue stream of the product over its life cycle (this is not about the technical performance of the product!)
  • Development project expense: one-time costs associated with the product development project

Development trade-off model

Fig. 2 – Create the Baseline Model (taken from: http://marketingsage.com/3-straight-forward-steps-to-calculating-roi-potential-and-making-better-numbers-based-decisions/ by David Lamont, April 2003)


Fig. 3 – Advantages and disadvantages of incremental innovation (from “Developing Products in Half the Time” by Donald Reinertsen and Preston Smith, p. 78)

Financial – advantages:

  • Earlier profits
  • Earlier cash flow
  • Less investment risk

Financial – disadvantages:

  • Duplicate fixed cost per product introduction

Marketing – advantages:

  • Shorter planning horizon
  • Earlier customer feedback
  • More reliable feedback
  • Flexibility
  • Image of consistent product improvement
  • Customer lock-in

Marketing – disadvantages:

  • Channel overload
  • Sales force overload
  • Customer overload
  • Service and support overload

Engineering – advantages:

  • Lower technical complexity
  • More motivating to teams
  • Early field experience with technology
  • Spreads technological commitments

Engineering – disadvantages:

  • Hard to make technology breakthrough
  • Boring products

Other – advantages:

  • Accelerated learning

Rapid prototyping – Storyboarding, paper prototyping and digital mockups

In a the online Human-Computer Interaction course that I did recently, I learned quite a bit about “rapid prototyping”. As the name suggests rapid prototyping is all about mocking up ideas quickly, to get user feedback fast.

In the lecture on this topic, Scott Klemmer – an Associate Professor of Cognitive Science and Computer Science & Engineering at UC San Diego – started off by explaining that degree of prototype fidelity is likely to evolve over time (see Fig. 1 below). One might start with a simple drawing on a piece of paper and eventually create a full-fledged clickable prototype. The goal, however, remains the same: to get quick feedback on an idea or concept. Prototyping is a great way to create and to compare alternatives. The idea is not to create pixel perfect designs but instead to rapidly create a number of designs that one can quickly evaluate and compare.

In the lecture, Scott outlined the following prototyping techniques to get quick feedback:

  1. Storyboarding – The main purpose of storyboarding, as Scott pointed out, is to focus on the task that the user interface (‘UI’) is going to support. The temptation is often to jump into sketching a user interface straight away, but there’s a lot of value in taking a step a back to concentrate first on what the UI will help the user to accomplish. Ultimately, one can look at storyboards as a communication tool that will help to convey flow and ideas. Scott provided a good outline of both the objectives and benefits of storyboarding, which I’ve copied in Fig. 2 below.
  2. Paper prototyping  Paper prototyping is a quick and easy way to figure out the UI at an early stage. One can use paper prototypes to quickly test multiple prototypes simultaneously, even getting the users involved in modifying these prototypes. Scott also provided some practical tips and tricks with regard to doing paper prototypes (see Fig. 3 below).
  3. Digital mockups – Naturally, with digital mockups the fidelity is likely to be a lot higher compared to storyboards or paper prototypes. However, this also means that creating digital mockups can be much more time consuming. I’ve got good experiences with observing users play with clickable prototypes or letting them complete specific tasks through the mockup, but this approach often requires a lot more planning and resource compared to other rapid prototyping techniques mentioned above.

Main learning point: I guess the main thing I learned from people like Scott Klemmer and Bill Buxton (see Fig. 4 below) is that with rapid prototyping the level of fidelity hardly matters. If anything, prototypes can serve as great communication and learning tools. Whether the goal is to quickly compare a number of alternatives or to get feedback on a particular idea, prototypes can really help in fleshing something out and getting rapid (user) feedback.

Fig. 1 – The level of prototype fidelity is likely to evolve over time (taken from http://tisquirrel.wordpress.com/2013/05/17/prototyping-storyboarding-paper-prototyping/)

Time - fidelity

Fig. 2 – Objectives and benefits of storyboarding (by Scott Klemmer)

Things that storyboards can help to convey:


  • Who are the people involved?
  • What is the user environment?
  • What is the task(s) being accomplished?


  • What steps are involved?
  • What leads to someone using the app?
  • What task is being illustrated?


  • What motivates people to use this system?
  • What does it enable people to accomplish?
  • What need does the system fill?

Benefits of storyboarding:

  • Holistic focus, concentrating on user tasks
  • Avoids commitment to a particular user interface (no buttons yet)
  • Helps to get all the stakeholders on the same page in terms of the goal

Fig. 3 – Some tips and tricks in relation to paper prototyping (by Scott Klemmer)

  • Keep all your materials in one place
  • Work quickly and make reusable components (e.g. buttons)
  • If something is too difficult to simulate, just describe the interaction verbally
  • Backgrounds can provide valuable context to the user
  • Don’t be afraid to mix and match software and hardware

Fig. 4 Bill Buxton on sketching experiences (at the Institute of Design Strategy Conference, May 2008)


Related links for further learning:

  1. https://ccrma.stanford.edu/courses/250a/lectures/IDSketchbok.pdf
  2. http://www.designinginteractions.com/interviews/BillVerplank
  3. http://www.uxbooth.com/articles/paper-prototyping/
  4. http://www.paperprototyping.com/
  5. http://en.wikipedia.org/wiki/Paper_prototyping
  6. http://www.slideshare.net/toddwarfel
  7. http://hci.stanford.edu/courses/cs247/2012/readings/WhatDoPrototypesPrototype.pdf
  8. http://hci.stanford.edu/publications/bds/10-Schrage.pdf
  9. http://prototyping-research.tumblr.com/post/35663689596/technique-video-prototypes
  10. http://citeseerx.ist.psu.edu/viewdoc/download?doi=