Why lifting the lid on black box AI is essential for healthcare adoption
Dr Ameera Patel, CEO of TidalSense, an AI respiratory healthtech pioneer for COPD and asthma solutions, explains why the patient safety risk posed by inscrutable AI algorithms cannot be overlooked in the quest for sustainable healthcare solutions.
Artificial intelligence (AI) and deep learning models have been around for decades, but it is only recently that computing power has enabled a step-change in the affordability and accessibility of these technologies.
And it is some step-change. Today, AI is being embedded into everything from kitchen appliances to cars. Even my Oral-B toothbrush is supposedly AI-enabled.
Dental hygiene aside, this ubiquity is not reflected in the healthcare sector, where AI uptake has been far slower, and for good reason.
The reticence among clinicians and regulators is not surprising – if algorithms are making life-or-death decisions, it's appropriate to treat them with caution, rigour, and healthy scepticism. This scrutiny is perhaps even more apt given that several AI technologies regulated by the Food and Drug Administration (FDA) in the US, rely on 'black box' methods of deep learning, whereby the decision-making (output) is almost impossible to trace back to the input, with limited, or no, ability to understand how the decision was reached.
However, the pressure to embrace AI is mounting as healthcare systems struggle with staff shortages, strikes, and pay disputes, the growth of chronic diseases and the challenge of ageing populations.
It is widely acknowledged that the current situation is unsustainable - particularly given the growing concerns that organisations are now so understaffed that they have little to no time to provide proper human care and compassion.
AI is needed to alleviate the strain on clinicians, nurses, and other health practitioners. But can black box AI ever be trusted to play a meaningful role in one of the most time-consuming aspects of healthcare provision: diagnosis?
Learning the right features
Diagnosis, whether performed by humans or AI, is all about pattern recognition – estimating likelihood based on the patterns presented.
Pattern recognition is how a clinician knows that the patient in front of them almost certainly doesn't have a super rare condition, despite displaying 90% of the symptoms, because they have also seen thousands of more common conditions which present in a similar way.
AI algorithms go through the same process but at far greater speed. They can be trained to assist clinicians in diagnostic decision-making, to speed up triage and prioritise patients, or to review medical imaging to determine which scans should be investigated further.
In each instance, the algorithm uses pattern recognition to arrive at a decision. The better the data quality used for training, the more accurate the AI's decision, provided, of course, that the algorithm has learnt the correct 'features' of each pattern. However, any bias in the data used to train the algorithm will result in incorrect decisions: ‘rubbish in, rubbish out’.
Hence, when used in a healthcare setting, any AI in use must give a clear and accurate answer and show how it got to that answer.
The truth about cats and dogs
Unfortunately, in the case of black box AI, there is scant visibility into this process. Most deep learning models fall into this box.
To illustrate the point, as humans, we know a cat is a cat because we recognise certain features about a cat's appearance that are characteristic of it being a cat.
However, it has proved possible to fool some AI algorithms into misinterpreting patterns and identifying cats as dogs by simply adding some imperceptible noise to the picture – a horrendous error when judged according to human pattern recognition, but one that could have a perfectly logical explanation when made by an algorithm.
Suppose that, rather than learning the characteristics humans associate with cats, the algorithm has determined that, in the pictures of cats it has seen, the cat is positioned slightly further to the left than in the images of the dogs. Consequently, it then starts to classify every object positioned to the left of an image as a cat and every image positioned to the right as a dog. Cats now become misidentified as dogs with disturbing frequency.
It's an extreme example, but it highlights the potential risks of using black box AI solutions. Even where the AI is still subservient to human decision-making, clinicians could easily be thrown off by an erroneous and seemingly inexplicable AI diagnosis, unsure of whether it's the algorithm at fault or they've genuinely missed something.
And in the case of autonomous AI – i.e. decision-making without supervision – trust and interpretability are utterly fundamental. Get the technology right, and it could be transformative within healthcare, for example, diagnosing huge volumes of patients at speed to reduce waiting lists. Get it wrong and, not only will patient safety be compromised, but clinicians will end up wasting far more time on over-investigation and correcting errors.
More transparency and collaboration are needed in AI development
Thankfully, the NHS has recognised the risks posed by black box AI, and its technology adoption guidelines increasingly state the requirement for AI models to be interpretable. On this basis, it seems highly unlikely that Generative AI such as the much-hyped ChatGPT will be adopted into the British healthcare system any time soon. Any AI like this that has billions of inputs is not, by definition, easily interpretable.
Likewise, the FDA is also getting stricter about black box AI, asking more questions about overly complex technologies before granting approval.
Some may argue that this shift away from black box technologies stifles the potential of AI. I view it as a necessary trade-off for enhanced patient safety, that places the onus back on AI developers to build more robust, transparent and trustworthy solutions.
Limiting the use of black box AI will force solution providers to better engage with industry during the AI development process. Clinicians need to help developers understand what 'explainable' means in practice – in other words, which aspects of the AI solution's decision-making process do they need to see/understand?
AI algorithms show considerable promise in diagnostics, and while we should not view them as a replacement for human decision-making in healthcare, many are already being piloted across the NHS to automate the detection of disease patterns, to give more time back to practitioners, and improve the quality of patient care.
Transparency, however, is a prerequisite for mass adoption in such a heavily regulated industry. And AI solutions need healthcare experts to help develop the algorithms – until this is the case, clinicians simply won't trust them.