The new FDA?

The FDA recognizes that it's limiting medical device innovation, and slowing the release of critical advances in medicine. The FDA now classifies some software as a medical device. Current software practices have short development cycles, whereas the FDA does not.

The FDA is working to revolutionize its processes, to promote innovation and speed-up delivery of new medical advances. Last week, I participated in a 2-day workshop - "Public Workshop - Fostering Digital Health Innovation: Developing the Software Precertification Program" - designed to hash out the details of how companies will be certified to play an active role policing the release of medical devices, and to explore early learnings of their new pilot program.

Pre-Cert, for short, is in a pilot phase with nine companies who are collaborating with the FDA to define how such a program would work. The goal is to allow a company to apply for Precertification and, once approved, they would have streamlined access in submitting applications for new devices and be able to revise their systems expeditiously. The FDA has made it clear that it is the entity being precertified, not the device. Clinical evidence will still be needed. But there will be new efficiencies in the application process enabled by this program.

The keynote presentation was given by Jeffrey Shuren, MD, JD the Director of the Center for Devices and Radiological Health (CDRH) at FDA. He talked about where the FDA is headed, and explained the point of this program. The goal is to lower barriers to innovation, speed time to market, but while maintaining high standards for quality and safety. The medical device regulation system was created over 40 years ago; it was developed for hardware. It has never been defined for software.  Software now has short timelines; products are improved on the timescale of weeks. The current FDA processes can not react at this speed. This is part of why this new FDA endeavor starts with SaMD.

SaMD, SiMD...

Software as a Medical Device (SaMD) is where a medical function is fulfilled only through software; Software in a Medical Device (SiMD) has both hardware and software components.  The Pre-Cert program will start only with SaMD.  There were frequent requests on the need to extend this, to at least SiMD products.

Bakul Patel, Associate Center Director for Digital Health, explained that he intends to focus only on SaMD initially; He plans on having the systems in place by the end of 2018 to start precertification, and he says this is not achievable unless focus is maintained.  After initial success has been demonstrated, the scope will be expanded.

The FDA recently release draft guidance on SaMD evaluation: Software as a Medical Device (SAMD): Clinical Evaluation.

Pre-cert member companies

Representatives of nine of the initially-selected members took part: Apple, Fitbit, Johnson & Johnson, Pear Therapeutics, Phosphorus, Roche, Samsung, Tidepool, and Verily. Tidepool has 13 employees; Apple has 10,000 times more (> 123,000). The FDA purposefully chose a diverse group of participants, to maximize learning from the pilot program.

The FDA made it clear that this new endeavor represented a new paradigm, and that this creation could only be done collaboratively.  When we would ask a question of the FDA, the frequent answer was "how do you think it should be?," or "what would work for you?"

Each of the member companies talked about their experience; unanimously they reported a sense of collaboration with the FDA.  Many expressed their surprise at this - it is different from the flavor of interactions in the past.  This was one of the things which left me feeling that the FDA might be able to pull this off - a new FDA, collaborative and responsive, seeking solutions using a grass-roots model.

Co-creating the program

The agenda read: "Day 2 - Co-Creating the Program."  It became clear what they meant.  We were put in smaller groups, with each group given a piece of the puzzle.  It really felt like co-creating - the FDA was highly engaged, but most of the legwork was done by the 500 representatives from potential pre-cert companies and other stakeholders: trade groups, payers, providers, and hospitals.

Some of the themes we dove into included:

• Drivers of Excellence and Associated Metrics
• Excellence in Results and Outcomes
• Aggregation Tools and Scorecards

Something we need to sort out is how can we assure excellence of the company being certified?  What are the metrics that would enable us to track that? What are the systems companies could implement to guarantee excellence?

And a theme throughout the discussions - how can all of these be made to apply both to tiny and to behemoth companies?

Systems and excellence

To be precertified, a company will need to demonstrate excellence.  Part of how this will be achieved by it demonstrating it has systems in place that will maintain excellence, and also that will provide alerts where excellence isn't there.  This leads to hard topics - how is excellence defined?  How will it be monitored?  What kind of systems need to be in place?  What is the difference between the kinds of systems needed for a start-up vs for a large enterprise? 

These are the kinds of questions we explored, and there were more questions than answers. The FDA will be busy, between now and the end of the year, when it hopes to finish design of the Pre-Cert program.

Will this work for a small start-up?

It is a goal of the FDA for this process to be inclusive, part of why small startups were part of the pilot program. Through the workshop people talked of the desire for a level playing field.  But is this realistic?  In one of my group's breakout sessions, we were asked to identify if there should be any criteria for who should be included in the program. The discussion was by no means conclusive, but some of the ideas were that there should be different levels of members, corresponding to certification in handling a certain risk level. There was discussion on whether or not the company should already have a prototype.  The certification process is to certify the entity, not the products.  We thought we shouldn't waste taxpayer dollars to attempt to certify companies which weren't viable companies. We came up with the idea of a pre-precertification, where they would submit a form to demonstrate their level of development, to demonstrate they were worthy to apply for precertification.

For me, the key question is this: for a startup to be successful, focus is necessary.  For most startups, this mean they will launch with a single product.  So for precertification to work for a startup, it's just a math problem: is the effort needed to achieve precertification plus the effort needed to achieve approval of this device less than the effort in the normal process to approve the device?  So, what is the scope of the reduction in the device application process for a pre-cert company?

I asked Bakul Patel about this.  His answer was that application to the program was purely voluntary, and that the goal was to be inclusive.  I'm not sure that addressed my question.

Data and machine Learning

In my group I brought up the issue of data - including both the derived and the raw data. This resonated with the group, and we thought this should become one of the main pillars of the program. Data is increasingly critical to the development of new products. If the data quality is bad (it's noisy, it's not well labeled, etc.), this could degrade the resulting product. Creating systems to monitor quality of data will be key to assuring quality of the company and of the product.

We didn't delve into Machine Learning. This will be an important area to explore in depth. Understandable AI will play an important role. This area is very much in the public eye in Europe, but in the US is meeting with resistance and a lack of awareness. How do we approve a device consisting of an algorithm we don't understand?

KPIs

There was a big emphasis on Key Performance Indicators - what metrics should be used to monitor the quality of a company? Should there be universal KPIs, or should it be company-specific?  Having them be universal allows for comparing between companies. But is a one size fits all approach relevant? Will this discriminate against smaller companies?

For what will the KPIs be used? Continuous monitoring of quality was thought to be important. Will maintaining KPIs become a burden on program participants? Will companies avoid joining the pre-cert program out of fear of the represcusions of the FDA monitoring their KPIs?

Brexit

This process reminds me of the Brexit process. It represents an end of old ways of doing business, and particularly of a change of how relationships work.  Just as in the EU, new ways of relating are being created, and old practices are being abandoned.  There will be a necessary confusion as things are sorted out. New structures will be defined, which has everyone in dialog.  Dialog is good. Perhaps unlike with Brexit, everyone has their eye on the same prize. The prize is maintaining patient safety while streamlining processes, reducing regulatory burden, reducing time to market, and particularly reducing the iteration time of new releases.

There will be inevitable costs associated with increased speed and innovation, many of which would be hard to anticipate.  I think of the recent news item, where the Strava fitness app exposed military base locations through disclosure, on an aggregated level, of where its users are jogging. Who could have anticipated this?  Surprising issues are bound to arise in the new world being created.  The key, then, will be how agilely the systems being created will be able to identify, respond to, and correct for, such issues.

In “Black Box Thinking,” Matthew Syed defines the term “black box thinking,” and says “it is about the willingness and tenacity to investigate the lessons that often exist when we fail but which we rarely exploit.  It is about creating systems and cultures that enable organizations to learn from errors, rather than being threatened by them."  He contrasts two industries, aviation, and medicine.  Aviation has focused on improvement and learning from its mistakes, and has dramatically reduced accidents.  It meticulously investigates its black boxes when there is an issue. In medicine, putting in corrections for mistakes has not been done with the same rigor, and there not been a comparable reduction in deaths or adverse patient outcomes.  A study of this "put the number of premature deaths associated with preventable harm at more than 400,000 per year." A Johns Hopkins University professor "pointed out that this is the equivalent of two jumbo jets falling out of the sky every twenty-four hours." They aren’t analyzing their black boxes.  

This is where the opportunities lie with the new FDA – tracking the data that will permit us to learn from mistakes, and put in timely corrections.

How can we foster innovation, decrease overhead and maintain quality?

We had numerous discussions on this. The pre-cert program adds new processes, and thus new burdens, both on the individual companies and on the FDA. So, to make this program worthwhile, more burden will have to be removed from current processes than is added by the pre-cert program.  From where will the gains come?

People, and particularly Bakul, talked about advances through automation of the processes. Part of the win is pushing the burden of policing onto the precertified company.

Bakul has made it clear that the standards can't be reduced. The hope is that the new processes will allow the FDA to focus where the highest return on investment would be achieved.  Attention could be taken off lower-risk devices or processes, and focused on high-risk ones.

A key goal is to have the level of regulatory investigation proportional to the risk. So then a key question becomes: can we identify risk up front? Can we identify changes in risk? One speaker mentioned that in the development of the product, features are changing, which means the level of risk could be changing.  What processes does one set up to track and deal with shifting levels of risk during the development process?

Conclusions

As Jeff Shuren put it, the goal is simplicity: “stop doing things where the value is not added. That does not mean we're reducing regulatory standards.  It does mean we best use our resources.”

I was quite impressed by the clear evidence of the FDA’s collaboration with, and even reliance on input from the companies it regulates.

The success of pre-certification will depend on what can be cut out, what can be streamlined and simplified.  This will require a shift in culture, both on the part of the FDA and of the company involved.  Tim Anderson, president of Anderson Leadership, pointed out that culture shifts are not fast - they require 2-3 years. I'm curious how fast the FDA will adjust to the new pace and ways of doing business.  Clearly the core group responsible for this project is quite enthusiastic.  How effective will they as evangelists with the rest of the organization?

Is this a new FDA? This workshop represents an impressive start of creating the new program. Much work has yet to be completed before this program is rolled out by the end of the year.  I'm encouraged - the signs of a new FDA are there.  Their implementation, and their success with internal cultural shifts, will ultimately decide if this is a new FDA.