Data poisoning: your questions answered

It is the manipulation of pre-training, fine-tuning or embedding data to introduce vulnerabilities, backdoors or biases[4]. OWASP calls it an integrity attack, because tampered data undermines the accuracy of what a system produces[4].

In one study, 250 poisoned documents backdoored models from 600 million to 13 billion parameters, while 100 did not reliably do so[1]. The authors found the absolute count mattered, not the share of training data[1].

That is not yet known. The authors say it is unclear whether the pattern holds for larger models or more harmful behaviours[1]. They tested up to 13 billion parameters[3].

No. Poisoning changes the data a system learned from, while prompt injection manipulates it through input at run time. OWASP lists them as separate risks, though it notes weak filtering can let injected content become training data[4].

Only partly. OWASP says backdoors can stay dormant until a trigger activates them, which makes them hard to test for[4]. It suggests red-teaming, monitoring and checking outputs against trusted sources[4].

Researchers said in 2023 that about 0.01% of the LAION-400M or COYO-700M datasets could be poisoned for about $60[8]. They notified the maintainers of the affected datasets[8].

In a medical study reported by Hackaday, corrupted models were not readily caught by standard medical benchmarks[9]. Hackaday says replacing about 0.001% of training tokens was enough to produce erroneous medical statements[9].

The NCSC publishes guidelines for secure AI system development, first published on 27 November 2023[6], and machine learning principles at version 2.0 from 22 May 2024[7]. For poisoning controls in detail, OWASP's LLM04 entry is the most specific[4].

List every dataset, feed and document store your AI systems learn from or read, and track where each came from[4]. Then version the data and ask suppliers how theirs is checked[4][6].