AAMI has published a consensus report (CR) for identifying, evaluating, and managing risk for healthcare technology that incorporates artificial intelligence (AI) or machine learning (ML).
AAMI CR34971:2022, Guidance on the Application of ISO 14971 to Artificial Intelligence and Machine Learning, responds to an urgent, immediate need. Existing standards for regulated medical devices do not yet adequately address the potential risks of emerging AI and ML applications, which “could jeopardise patient health and safety, increase inequalities and inefficiencies, undermine trust in healthcare, and adversely impact the management of healthcare,” the CR states.
Now, for those familiar with the widely used international standard, ISO 14971:2019, Medical devices—Application of risk management to medical devices, AAMI’s CR is a must-have companion for risk management of AI- or ML-enabled medical systems and devices.
Pat Baird, co-chair of the AAMI Artificial Intelligence Committee and senior regulatory specialist at Philips, said: “We intentionally structured the CR to be easy for people that know 14971 to use. Readers are probably aware of the existing companion document, 24971, which provides guidance on how to use 14971. We modelled the structure of 34971 to be similar to a section in 24971 about risk management for in vitro diagnostics. The idea is that the risk management process is the same, and here are a few new ways that this particular technology can fail that you might not have thought about.”
Baird is part of a small task force of the AAMI Artificial Intelligence Committee that developed CR34971, which was reviewed by the full committee and by risk analysis experts at the British Standards Institute (BSI). The committee then approved the consensus-driven report. AAMI and BSI plan to use this CR as the basis for an AAMI technical report and a British Standard.
Joe Lewelling, senior advisor on content and strategy at AAMI, said: “We hope to complete the AAMI technical report and the BSI standard sometime this year. The AAMI committee, which includes clinical, manufacturing, regulatory, information technology, and risk management expertise, is working hand in hand with a similarly focused BSI committee on these documents.”
Longer term, AAMI and BSI expect to propose these resources to the International Standards Organisation as guidance, informative, or annex documents to ISO 14971 or ISO 24971.
Learning from other industries
Baird said: “To develop the CR, we conducted a literature review for ML failures in multiple industries, in an attempt to learn from others that have gone before us.”
Additionally, the task force reviewed prepublication documents from ISO/IEC JTC 1 / SC 42, a subcommittee of a joint technical committee on artificial intelligence that is developing a series of horizontal (cross-sector) standards that address such issues as bias management.
The CR offers insights into how risk management systems and processes can be adapted for AI and ML medical devices. It also details safety-related characteristics and considerations in five areas:
- Data management
- Bias
- Data storage, security, privacy
- Overtrust
- Adaptive systems
The CR includes annexes covering the risk management process, risk management examples, considerations for autonomous systems, and personnel qualifications.
For example, personnel qualifications apply to people developing AI- or ML-enabled products.
Baird said: “One of the things we noticed in the literature about ML systems is that many times failures occurred because, although the development team had data, they didn’t have knowledge. Developers had logical assumptions regarding the use of their product, but the reality was different, leading to failure. To be successful, we really need to understand the context of use and leverage the wisdom around us. We felt it was important to stress this point when discussing risk management.”
Next Steps
The AAMI Artificial Intelligence Committee is now turning its attention to other issues that warrant exploration and consensus, such as change control for systems that continue to learn over time.
Baird said: “The use of ML in healthcare has the potential make significant improvements in the delivery of healthcare, but only if those products are safe and effective. So, the logical first step is to address safety-related concerns. New technologies always introduce new risks, and good risk management is obviously a key driver for the medical device sector. We always put safety first, and improvements to performance can come a little later.”
