Remote testing has become a hot topic of discussion within the medical product development fraternity. This is perhaps no surprise given the well-documented challenges imposed on us all by the COVID-19 pandemic. It’s a concept already being explored by many in the industry, but there’s no doubt that remote testing presents many logistical, procedural, data security, confidentiality, and safety hurdles. And for me, the big question is this: it may be a viable option, but will the regulators accept it?
Creating engaging, safe & effective experiences for consumers, patients & healthcare providers alike.
Human Factors Engineering (HFE) is a vital component of successful medical product development. Gaining insights from real users from the outset and throughout the design process – in the form of their needs, wants or their ability to use a product correctly – is crucial. One of the key methods of eliciting these insights is an HF study, also known as a usability study or user testing. Usually this involves an HF expert acting as a moderator with an intended user as a participant of the study. But as I write, conducting a face-to-face usability study, such as a formative or validation study, is extremely challenging.
Our current plight has forced medical product manufacturers and HF specialists alike to think innovatively about new ways to gather HF data from real users – and thrown the spotlight on remote testing. The topic was discussed last month by many presenters, including some from the US Food and Drug Administration (FDA), at the virtual International Symposium on Human Factors and Ergonomics in Health Care, organised by the Human Factors and Ergonomics Society (HFES). The event was attended by hundreds of HFE professionals, medical product manufacturers and regulators such as the FDA and Health Canada.
Challenges and questions
Remote testing is a way of conducting an HF study or testing where the participants sit at one end of a web conferencing link, usually at home, and the moderator and observer sit at the other. Remote testing is not new to the HF and market research communities and has been used as a tool to test Mobile Medical Applications (MMAs) and web-based applications for a long time. But its utilisation for physical medical products, especially for regulatory submissions, has been limited. While the physical separation between the participants and researchers is the key driver for remote testing in the COVID-19 situation, this separation naturally poses many challenges and a host of questions.
Would representative users of the product being tested have enough knowledge of and confidence in using a web conferencing platform? How would you find and recruit representative users? Is it viable to ship multiple cameras and device prototypes to every participant? Would every participant have reliable internet/wi-fi connection at home? Can a moderator run through a discussion guide over the web conference without compromising the natural flow of the use process and without being too intrusive and leading? Can the participants take the ‘non-leading’ instructions from the moderator and simulate the use process in a realistic manner; and can the observer capture all the nuances of the use process, and more importantly, the subtle use errors and difficulties?
On the top of the logistic and procedural challenges, there are pertinent risks of data security, confidentiality and participants’ safety. What if, for example, someone commits a use error that poses them a safety risk? Remote testing for medical products certainly demands the most careful consideration. And when it comes to its acceptability to regulators? The question was put directly to FDA representatives at the symposium, but the answer is not as straightforward as we’d perhaps like.
FDA’s current perspectives
The event was attended by personnel from the Center for Devices and Radiological Health (CDRH) and Center for Drug Evaluation and Research (CDER) at the FDA. Things that the pandemic has not (yet) changed are the HFE process and the FDA’s review process and requirements unless the product qualifies for an Emergency Use Authorization (EUA). The FDA says it “recognises that the COVID-19 public health emergency may impact the feasibility and appropriateness of conducting in-person human factors (HF) testing of medical products due to any public health measures implemented to control the virus”.
However, the agency is sceptical about the appropriateness and acceptability of HF data collected via remote testing, especially for the final validation testing. This is because “the agency is currently not aware of any data that supports the use of remote HF validation testing or of any consensus scientific guidelines or standards that can inform as acceptable remote testing approach”. Indeed, the FDA personnel mentioned on multiple occasions that “the agency is unable to provide a general statement at this time about whether remote HF testing could potentially be an acceptable approach”.
A presentation from an HF reviewer at the CDRH helped synthesise the agency’s thought process behind this scepticism. There are broadly two aspects of the remote testing method – ‘representativeness of the real world’ and ‘completeness and accuracy of the data’ – that the FDA is concerned about. One representative questioned the remote testing with regard to its ability to represent the real world – test participants (e.g. tech savvy) vs intended users (e.g. lay users), test environment vs intended use environment, test device user interface (e.g. modified due to remote testing) vs final device user interface, and test scenario vs real use scenario.
He suggested that remote testing may not be representative, and so it may be difficult to utilise the method for a validation study. Similarly, it may be challenging to elicit and capture the necessary HF data – performance data, knowledge task data, root cause for use errors and difficulties, and subjective data – accurately and completely through the eyes and ears of a camera, hence compromising the data quality.
My final thoughts
With the limiting nature of the current technologies and facilities, it is likely that remote testing may not be appropriate for validation testing for many types of medical products. But it can be used for formative evaluations where the intention is to learn as much as possible to improve the design of a medical product. The appropriateness of remote testing for them was not discussed explicitly at the symposium. One reason for this is that formative evaluations are recommended but not mandatory. The FDA’s reviews primarily focus on results from validation studies.
Nevertheless, it is plausible to think that this method can provide some benefit to the design as opposed to doing no user study in the current COVID-19 crisis. The data may not be as rich but is still helpful. However, it is important to avoid any misleading data resulting from the shortfalls of this kind of testing.
In general, as discussed above, the regulators’ requirement for validation testing remains the same as before. Also, remote testing methods may not be acceptable for many medical products that require physical interaction with their user interface. However, as the regulators are learning and monitoring the situation closely, it cannot be ruled out that remote testing, or some other form of testing that does not require a face-to-face meeting, will become acceptable in one form or other if the COVID-19 situation continues.
In the meantime, the FDA strongly encourages medical product manufacturers (sponsors of the products) to engage with it as soon as possible and to “reach out to their project manager” for guidance on the HFE process, HF data requirements, especially the validation testing, for their specific product. If a sponsor is planning remote testing as part of the data gathering for their regulatory submission, they are encouraged to discuss the study design and protocol, along with samples of the products, with the FDA before embarking on the study.
To learn more about how remote testing fits within the wider HFE process and device development, refer to the author's article.
