Case study
Poisoned medical AI: a study of how little it takes
A study in Nature Medicine, as reported by Hackaday, found that replacing about 0.001% of training tokens with medical misinformation produced models that often gave wrong medical statements[9].
Last reviewed against Anthropic, UK AI Security Institute and arXiv. Sources
This is the clearest worked example of poisoning in a high stakes setting. The primary paper was behind a paywall when this site was prepared, so everything here rests on Hackaday's published account of it [9].
What was done
The experiment
The defence tested
Checking outputs
The researchers cross-referenced model outputs with biomedical knowledge graphs, flagging claims that could not be verified [9]. In a test on 1,000 random passages, Hackaday reports a claimed effectiveness of 91.9% [9].
The caveats Hackaday notes
The approach does not guarantee that misinformation gets past the graphs, and it does not solve the underlying problem that outputs cannot be fully trusted. Post-training adjustments and retrieval do not fix corruption from poisoned training data [9].
Lessons for other sectors
Beyond medicine
You do not need to run a clinical system to draw something from this. Any organisation using a model where wrong output has a cost, such as legal, financial or safety advice, should ask two questions. First, who could have altered the data? Second, what independent check exists on the answers?
OWASP also recommends checking model outputs against trusted sources [4].
Useful questions for a supplier:
- Where did the training data for this system come from?
- Which checks apart from benchmarks have you run?
- How are outputs checked against a trusted source?
Keep reading
Read next
Prevention
Practical controls to reduce data poisoning risk, drawn from OWASP and NCSC guidance: provenance, versioning, vetting and sandboxing.
Read the guide
Detection
How to look for signs of poisoned data: output checks, adversarial testing, monitoring and why benchmarks alone are not enough.
Read the guide
Guidance
Where UK, US and industry guidance covers data poisoning: the NCSC guidelines and principles, NIST's taxonomy and OWASP's LLM risk list.
Read the guideHow CyPro can help
Need a hand planning for this?
Understanding the change is one job. Working out what it means for your systems, your suppliers and your roadmap is several more. CyPro's consultants can take any of them on.
Fractional CISO for the board
A senior security lead who advises the board on what to fix first, what to accept and what to tell customers, and owns the decision record.
More on cypro.co.uk
Hardening and segmentation
Our consultants put the controls in place that limit the damage when something is missed: firewall rules, network isolation and tighter admin access.
More on cypro.co.uk
Penetration testing
A focused test of your systems and the controls around them, so you know they hold before an attacker finds out for you.
More on cypro.co.uk
Vulnerability scanning
Find every internet-facing system you run, including the forgotten ones, and see which are still on a vulnerable build.
See the scanning service
24/7 security monitoring
Our SOC analysts watch your systems day and night for the signs of an attack in progress: unexpected processes, new outbound connections, logins that do not fit.
More on cypro.co.uk
Incident response
If you may already be compromised, our responders contain it, preserve the evidence, find out what was touched and get you running again safely.
More on cypro.co.uk
Plan ahead
Want a plan for your organisation?
Defending against data poisoning starts with knowing which AI systems you run, where their data comes from and who owns the risk. CyPro can help set the governance, risk assessment and controls around that.