Explainer
What data poisoning means, in plain terms
Data poisoning is the deliberate manipulation of the data a system learns from. The aim is to degrade it, bias it or plant a behaviour that an attacker can switch on later.
Last reviewed against Anthropic, UK AI Security Institute and arXiv. Sources
An AI model learns patterns from examples. If someone alters those examples, the model learns the altered pattern. OWASP defines data and model poisoning as the manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases [4].
The idea
Three things an attacker may want
The effects OWASP lists fall into groups. Each one matters differently to a business.
A backdoor
A trigger in the input makes the system act differently from how its owner expects. OWASP says backdoors can lie dormant until the trigger appears [4].
A way in to other systems
OWASP notes that poisoning can expose downstream systems to exploitation, and gives authentication bypass, data exfiltration and hidden command execution as scenarios [4].
Where it happens
Three stages of a model's life
-
1
Pre-training: the model learns from general data. -
2
Fine-tuning: it is adapted to a task. -
3
Embedding: text is converted into vectors that a system searches.
Easy to miss
Why testing may not show it
A backdoored model may behave normally until its trigger appears. For that reason, OWASP recommends red-team testing and monitoring of training behaviour as well as controls on the data itself [4].
Not the same as prompt injection
Prompt injection manipulates a system through the input it is given while running. Poisoning changes the data it learned from, before the system is used. OWASP treats them as separate risks, although it notes that weak filtering can let injected content become training data [4].
Where next
How big is the risk?
See the 250 document study for what researchers have measured, and types of data poisoning for the routes in.
Keep reading
Read next
Attack types
The main ways training, fine-tuning and retrieval data can be poisoned, and how poisoned model files differ, with sources.
Read the guide
The 250 study
What the Anthropic, UK AI Security Institute and Alan Turing Institute study found about poisoning language models, and what it does not show.
Read the guide
Web datasets
How researchers showed that datasets crawled from the web can be poisoned cheaply, through expired domains and timed edits.
Read the guideHow CyPro can help
Need a hand planning for this?
Understanding the change is one job. Working out what it means for your systems, your suppliers and your roadmap is several more. CyPro's consultants can take any of them on.
Fractional CISO for the board
A senior security lead who advises the board on what to fix first, what to accept and what to tell customers, and owns the decision record.
More on cypro.co.uk
Hardening and segmentation
Our consultants put the controls in place that limit the damage when something is missed: firewall rules, network isolation and tighter admin access.
More on cypro.co.uk
Penetration testing
A focused test of your systems and the controls around them, so you know they hold before an attacker finds out for you.
More on cypro.co.uk
Vulnerability scanning
Find every internet-facing system you run, including the forgotten ones, and see which are still on a vulnerable build.
See the scanning service
24/7 security monitoring
Our SOC analysts watch your systems day and night for the signs of an attack in progress: unexpected processes, new outbound connections, logins that do not fit.
More on cypro.co.uk
Incident response
If you may already be compromised, our responders contain it, preserve the evidence, find out what was touched and get you running again safely.
More on cypro.co.uk
Plan ahead
Want a plan for your organisation?
Defending against data poisoning starts with knowing which AI systems you run, where their data comes from and who owns the risk. CyPro can help set the governance, risk assessment and controls around that.