Insider Brief
- Data poisoning is an AI supply chain risk where attackers manipulate training data to influence model behavior after deployment.
- The article examines how poisoned data enters AI pipelines, why standard model evaluations may miss targeted attacks, and how organizations can reduce exposure.
- Practical defenses include verifying model provenance, protecting training datasets, testing for targeted behaviors, and maintaining records such as AI bills of materials.
In February 2024, JFrog’s security research team scanned model files uploaded to Hugging Face, the largest public repository for AI models. They found around 100 malicious models sitting among the millions available for free download. Several executed arbitrary code the moment a developer loaded them. This happened despite Hugging Face having security scanning in place.
Most organizations sourcing AI models from public repositories had no additional process to catch it. They downloaded the model, it worked, and they moved on.
The broader risk comes from how modern AI systems are built and deployed. Firms are downloading pre-trained models or fine-tuning on third-party data and are inheriting risks they often cannot see. Data poisoning is one of the most technologically rooted forms of this risk and consequently, one of the hardest to identify.
This article covers what data poisoning is and how it works, how it enters the training pipeline, why standard model testing rarely catches it, and what practical controls exist at each stage.
What Data Poisoning is
A model learns by studying examples. When trained on millions of text samples, it builds patterns that help it understand relationships between words, concepts, and ideas. These learned patterns shape how the model responds to future inputs.
Data poisoning targets this learning process before training is complete. An attacker adds carefully designed examples to the training data to influence the model’s behavior. Once the model learns from poisoned data, the changes become part of its internal parameters and remain after deployment.
Several distinct techniques fall under this category.
Backdoor Attacks
These types of attacks involve embedding a trigger in the training data for example a specific phrase, token, or pattern. The model learns to associate that trigger with a particular output. On any other input, it behaves normally. On the trigger, it does what the attacker intended, which mostly involves generating harmful content, bypassing a safety filter, or something else.
Label Flipping
These attacks alter the labels attached to training examples. For example, a fraud detection model trained on mislabeled data may learn to treat fraudulent transactions as legitimate. The model can still appear accurate during testing unless the evaluation data includes the manipulated examples.
Clean-Label Attacks
These attacks are harder to detect because the labels remain correct, but the examples themselves are designed to influence how the model learns. The data may look normal during review while still shifting the model’s behavior in a specific direction.
Across all three techniques, the goal is similar – a small number of poisoned examples can influence a much larger training dataset, creating targeted effects that remain after training and may not appear in standard evaluations.
How it Enters the Training Pipeline
Large models are not trained on hand-curated data. They are trained on datasets scraped from public web pages, code repositories, forums, and document archives. The volume makes manual review impossible at any reasonable scale.
An attacker does not need to breach a lab’s infrastructure. They only need to place content in the right public location before it gets scraped. IBM X-Force’s Patrick Fussell described this to CSO Online:
“If we know the models are going to scrape Wikipedia every other week, all we have to do is be in that window. We can plant some bad data, and then we know that’s going to be ingested into the model.”
And the quantity required to poison the data is smaller than most teams assume. Research from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that injecting as few as 250 maliciously crafted documents can implant backdoors that activate under specific trigger phrases while leaving general model performance unchanged. That finding applied to specific experimental conditions and model architectures, but it demonstrates that poisoning does not require large scale data access to be effective.
Why Standard Testing Misses it
Model evaluation typically measures accuracy, coherence, and task performance on a benchmark dataset. A backdoored model can score normally across all of those.
Mithril Security demonstrated this in 2023 with its PoisonGPT project. The team modified a public GPT-J-6B model so it produced false historical information when asked specific questions. In other words, outside of those targeted prompts, the model continued to perform normally on standard benchmarks.
Standard benchmarks miss this class of attack because the attack is designed to survive them. Evaluating a model on general performance tells a firm whether the model is capable. But it does not tell them whether the model has been deliberately modified to behave in a specific, targeted way on specific inputs.
The Supply Chain Angle
A model reaches an enterprise through several stages, and each stage introduces a possible point of compromise:
- Training data collection: Scraped or collected datasets can contain manipulated examples before training begins.
- Pre-training: The model developer may unknowingly train on compromised data.
- Public release: Open models can be modified, repackaged, or redistributed after release.
- Fine-tuning: Additional training on community or third-party datasets can introduce new risks.
- Distribution: Model repositories and cloud APIs create additional points where users must assess trust and provenance.
The JFrog finding mentioned above is one illustration of supply chain risk, though technically distinct from behavioral data poisoning. Those models carried malicious executable payloads in their file formats, not modifications through training data corruption. Both represent AI supply chain risk through different mechanisms. The OWASP LLM Top 10 covers both categories, ranking training data poisoning among the highest-impact risks for organizations deploying language models.
On the data side, datasets collected from public sources or third-party providers may contain manipulated examples before they are used for training. A single poisoned contribution in a shared dataset could affect multiple models built from that data.
AI Insider’s coverage of the Mercor supply chain incident was a great example of showcasing how security issues in widely used open-source AI tooling can spread into enterprise environments before organizations identify the source of the compromise.
Fine-Tuning Exposure
Fine-tuning is where many enterprises first interact directly with the model training process. An organization takes a pre-trained model and adapts it using internal data such as customer conversations, documents, support logs, or product manuals.
This creates two possible exposure points.
If the base model was already poisoned, fine-tuning may carry that risk forward. Adapting the model to a specific domain does not necessarily remove hidden behaviors embedded during earlier training.
The fine-tuning dataset itself can also become an attack surface. Data without proper access controls, provenance checks, or protection from unauthorized changes may introduce new risks before training begins. This is similar to software supply chain issues, where compromised dependencies can affect downstream systems.
RAG Pipelines and Inference-Time Risk
Retrieval-augmented generation (RAG) systems introduce a related but separate security risk. A RAG pipeline retrieves documents from a knowledge base at query time and passes them to the model as context.
Unlike data poisoning, this does not involve changing the model itself or its training data. Instead, attackers can place manipulated content in the retrieved documents and influence the model’s responses. This technique, known as indirect prompt injection, can affect model behavior without access to the underlying system.
Organizations using RAG systems should treat their document sources as part of the security boundary. Controls such as content provenance checks, access restrictions, and monitoring for malicious patterns can help reduce this risk.
What Organizations Can Do
Most of the practical controls here extend existing software supply chain security practices rather than requiring new programs from scratch.
Know Where Models Come From Before Using Them
Check for a published model card documenting training data, methods, and known limitations. Confirm whether the source repository provides integrity guarantees. Models without documented provenance carry risk that benchmark scores cannot reveal.
Apply Software Supply Chain Practices to Training Data
Training and fine-tuning pipelines need similar protections to software development pipelines. Access controls, audit logs, and integrity checks can help prevent unauthorized changes to datasets. A compromised training dataset can create a supply chain risk, similar to vulnerabilities introduced through compromised code dependencies.
Test for Targeted Behavior, Not Just General Performance
Red-teaming can help identify poisoned behavior by testing the model with sensitive topics, unusual prompts, and inputs outside its normal operating conditions. This approach focuses on understanding what the model should avoid, not only measuring what it can do.
Document the Model Supply Chain
A model bill of materials (AIBOM) tracks where a model came from, what data was used to train it, and how it has been modified over time. Without this record, organizations may struggle to identify where a model’s security risks were introduced.
Maintaining an AIBOM is necessary to help organizations identify where risks were introduced because it gives teams a clearer view of model provenance and helps them assess risks before deploying or updating AI systems.
For readers looking to learn more about AI security, AI Insider’s coverage of prompt injection, the attack type that exploits how AI systems process instructions – is a useful starting point, alongside AI Psychosis and how AI-powered scams and phishing attacks are evolving.
