Supply chain
Poisoning datasets built from the web: what was shown
A 2023 paper showed two practical ways to poison datasets that are assembled from web content. The weak point is the gap between when data is checked and when it is downloaded.
Last reviewed against Anthropic, UK AI Security Institute and arXiv. Sources
Many training sets are lists of links, not stored files. Whoever downloads them later gets whatever those links serve at that moment. The paper "Poisoning Web-Scale Training Datasets is Practical" set out to test that weakness [8].
Two attacks
Split-view and frontrunning
The authors, led by researchers including Nicholas Carlini, introduced two attacks and said they could poison 10 popular datasets [8].
Split-view poisoning
This exploits content that changes over time, so that the view a curator checked differs from what a later client downloads [8].
Frontrunning poisoning
This targets datasets that take periodic snapshots of crowd-sourced content such as Wikipedia, and needs only a time-limited window to inject malicious examples [8].
The numbers
Scale and cost
The paper states that about 0.01% of the LAION-400M or COYO-700M datasets could be poisoned for about $60 [8]. The authors notified the maintainers of each affected dataset and recommended several low-overhead defences [8].
Dated, but still relevant
The first version appeared on 20 February 2023 and the current version was revised on 6 May 2024 [8]. Check any dataset you use against its maintainers' current guidance, as protections may have changed since.
Why it matters to you
Dependencies you may not see
If a supplier or open source project trained on a crawled dataset, you inherit that dataset's exposure. The NCSC guidelines place supply chain security within secure development [6]. Knowing where your data comes from is the first step to managing that.
Ask these of any dataset you rely on:
- Is it stored, or is it a list of links fetched later?
- Does it include crowd-edited sources that can change between snapshots?
- Is there a way to confirm what you downloaded matches what was checked?
Keep reading
Read next
Medical case
A reported Nature Medicine study on poisoning medical language models with 0.001% misinformation, and why benchmarks did not catch it.
Read the guide
Prevention
Practical controls to reduce data poisoning risk, drawn from OWASP and NCSC guidance: provenance, versioning, vetting and sandboxing.
Read the guide
Detection
How to look for signs of poisoned data: output checks, adversarial testing, monitoring and why benchmarks alone are not enough.
Read the guideHow CyPro can help
Need a hand planning for this?
Understanding the change is one job. Working out what it means for your systems, your suppliers and your roadmap is several more. CyPro's consultants can take any of them on.
Fractional CISO for the board
A senior security lead who advises the board on what to fix first, what to accept and what to tell customers, and owns the decision record.
More on cypro.co.uk
Hardening and segmentation
Our consultants put the controls in place that limit the damage when something is missed: firewall rules, network isolation and tighter admin access.
More on cypro.co.uk
Penetration testing
A focused test of your systems and the controls around them, so you know they hold before an attacker finds out for you.
More on cypro.co.uk
Vulnerability scanning
Find every internet-facing system you run, including the forgotten ones, and see which are still on a vulnerable build.
See the scanning service
24/7 security monitoring
Our SOC analysts watch your systems day and night for the signs of an attack in progress: unexpected processes, new outbound connections, logins that do not fit.
More on cypro.co.uk
Incident response
If you may already be compromised, our responders contain it, preserve the evidence, find out what was touched and get you running again safely.
More on cypro.co.uk
Plan ahead
Want a plan for your organisation?
Defending against data poisoning starts with knowing which AI systems you run, where their data comes from and who owns the risk. CyPro can help set the governance, risk assessment and controls around that.