Unlocking the value of an expert review

In this three-part series, I want to explore the expert review. The expert review is often looked down upon. However, it can be one of the most powerful tools in the human factor engineers (HFE) toolbox. Some companies use the expert review as a foundation for projects while other companies seem to rarely, if ever, utilize the expert review.

In this first installment I want to explain why, for the value, the expert review can be one of the most important tools available to a human factors engineer. Part two will detail my process of creating a template for conducting expert reviews. Finally, in the third part, I will evaluate my template by conducting an expert review to see how well it works and discuss improvements and future considerations.

What is an Expert Review? 

An expert review is when an experienced human factors practitioner evaluating a system or device for the overall usability. As I see it, an expert review is the combination of two other types of analysis: a heuristic evaluation and ergonomic assessment.

What is a Heuristic?

For those of you who did not take intro to Psychology in college, heuristics are often described as a rule of thumb or generalization that our brain uses to quickly and effortlessly interpret the world or solve a problem. Considering the amount of stimuli our brain receives and the complexities of each situation, if we had to fully process all of the information to reach a logical conclusion it would take an exorbitant amount of time and energy.

Our brain is economical when it comes to exerting effort and will do everything it can to reduce the energy it spends. As an alternative to effortful thinking, the brain uses the heuristics as a way to understand and interpret the world. While these heuristics are not always correct, just like a rule-of-thumb, they provide a baseline interpretation that is often good enough.

A rule-of-thumb that many of us hear as kids is: do not talk to strangers. This is often a prudent rule and an effortless way for kids to know if it is okay to talk to someone. This takes the situational thinking away from the kids and makes it a simple if-then scenario. However this rule-of-thumb is not always true, whether you are talking to a doctor for the first time or the distant cousin at the family reunion that you have never met before, some “strangers” are okay to talk to. While not always correct, this rule allows effortless and quick decisions. Similarly, the time and effort we save from interpreting the world according to heuristics make the occasional errors worthwhile.

While we always have the option to critically assess situations, there are often instances when we do not have the time, do not have the resources, or do not have the motivation to. For these situations, heuristics are a necessary and important tool for the brain to utilize.

What is a Heuristic Evaluation?

A heuristic evaluation is a way to evaluate products based on how well they conform to certain heuristics. While there are many heuristics, it would be impractical to evaluate the system against all that are available.

Luckily for us, there are certain subsets of heuristics that are more relevant for certain industries. For example, one of the common lists of heuristics to use for medical devices in regards to safety was created by Zhang. This list comprises of 14 heuristics that each system should be assessed against. For user interfaces, the gold standard is the list of 10 heuristics from Nielsen.

The main goal of the evaluation is to assess products through the lens of heuristics. We know the brain will most likely assess the system with heuristics, so it is only logical to create devices that work under those conditions.

In my biased opinion, this is one of the key reasons the field of human factors even exists. Often devices are created by engineers who assume the system will be used while the brain is cognitively active, logically thinking through the process, and arriving at the correct end result. What engineers often fail to consider is the human variable, such as when the brain jumps to illogical conclusions or incorrect actions based on heuristics.

What is an Ergonomic Assessment?

An ergonomic assessment is evaluating a system for effective physical interactions. While the heuristic evaluation is all about cognition, the ergonomic assessment is about how well people will be able to manipulate and interact with the system.

Let's take a car steering wheel as an example. If we are testing a new car and when you turn the steering wheel right the car goes left, the heuristic evaluation would flag that as problematic. Not only is it inconsistent with a standard car on the market today, but it is also inconsistent with our internal considerations for how objects should move. However, the ergonomic assessment would instead be concerned about the interaction between the wheel and the driver, such as the force required to turn the steering wheel or how comfortable the steering wheel is to hold. Ergonomic assessments can also include topics like lighting, size of font, strength, grip, size, comfort, etc.

The focus of an ergonomic assessment will vary greatly depending on the device and industry. Someone evaluating a cockpit of a fighter jet might be concerned with the comfort of the seat, the field of view of the instrument panel, legibility of the instruments in various lighting conditions, or the tactile feel of the various buttons. However, someone evaluating an autoinjector for patients with arthritis is worried about very different issues, such as the force required to activate the injection, the finger dexterity required to remove the cap, or the precision required to replace the needle cap to avoid accidental needle sticks.

What is the Secret to Unlocking Their Value?

An expert review combines the heuristic evaluation and the ergonomic assessment to assess the overall usability of the system. The review can be conducted by one person or multiple people depending on the time and budget available. The review can focus on a high level tasks and subsystems or can be detailed enough where every single interaction is evaluated. The expert review is a framework that can be adapted to fit the specific requirements of a project or system being evaluated.

I do not want to mislead anyone with the title, so I will provide the key to understanding how to best utilize expert reviews. It is really simple in fact. Conduct them. Do them. Perform them. Plan them into your projects and utilize the results.

For the time and effort required, I would be hard pressed to find something that provides a greater value. The expert review is beneficial even if the HFE is not an expert in the specific field or system being evaluated. However, the benefits are even more extreme when the system is being reviewed by someone who has extensive experience within the field or with similar types of systems.

Put an autoinjector or pen injector in front of me and I will not only compare it to multiple devices on the market, but I will also confidently indicate where I think there will be usability issues. This does not mean I will be able to predict every usability issue or even always be correct in my assessment; however, I will be able to use the framework of the expert review to assess where issues may arise based on heuristics and ergonomics.

Before you go ahead and conduct that full scale user study, consider scheduling an expert review. With one person it can be fast and cheap and with 3-5 people independently conducting the review it can be quite comprehensive while still being time efficient. Expert reviews are a great way to catch and address the low-hanging fruit before investing in user studies.

At this point you may be saying to yourself: “If I have a HFE on the development team, there must not be any reason to do an expert review since the low-hanging fruit would have already been addressed.” While that is often the case, it is not always true. Someone who has been working on the development team has a very different role than an independent reviewer. Often the HFE on a project is required to make difficult decisions, due to technical constraints, budget limitations, schedule issues, client input, etc. All of those aspects, specific to the project landscape, influence the work the HFE does. However, the independent reviewer conducting the expert review does not care about any of that. Coming in with a fresh set of unbiased eyes, they are impartial and only care about assessing the the overall usability of the system.

There is nothing more wasteful and demoralizing than conducting a user study with a system that has usability issues which could have been discovered and addressed internally. For the value they provide given the effort required, especially compared to some other more commonly used tools in the HFE toolbox,  I cannot stress highly enough the benefit provided by conducting expert reviews.

Ben Zwillinger

Ben is a Human Factors Engineer. He has a background in psychology and neuroscience.