Poison-Proof Your AI With These 5 Steps
As far as cyberattacks go, data poisoning is right on brand. It just sounds bad, especially when the business world increasingly treats data as the digital equivalent of a precious metal.
That computes, because data poisoning is bad. The term refers to an emergent risk to AI models in which an attacker – who can be external or internal – deliberately corrupts training datasets to impact the model’s operations or outputs. That could include adding bad data, manipulating existing data, or deleting clean data.
If you’re building AI-driven SaaS products – say, a marketing automation app that includes an LLM tool – you need to take a closer look at how you’re protecting your models.
In this article, I’ll share expert advice on core strategies for minimizing the risk of data poisoning in your AI training datasets.
What is Data Poisoning?
An example of data poisoning courtesy of NIST, which, like many other security-minded organizations, is clearly paying attention to the threat: “[Attackers can slip] numerous instances of inappropriate language into conversation records, so that a chatbot interprets these instances as common enough parlance to use in its own customer interactions.”
Suddenly, your trusty customer service chatbot is using profanities (or even worse language) in conversation with your actual human customers. That’s probably not what anyone has in mind when they talk about using AI to boost productivity, efficiency, or innovation.
Regardless of use case, CTOs embedding generative AI or other AI-based applications into their SaaS products need to take steps to protect their training data from malicious additions, deletions, or other manipulations that can affect everything from model performance to user experience to brand reputation and more.
Also: experts generally seem to agree that prevention is the best posture with data poisoning. It’s usually easier to keep it from happening than to mitigate the impacts of an incident.
How to Combat Data Poisoning – 5 Principles to Minimize Risks
1. Know your data.
Strengthening your defenses against data poisoning begins with understanding what you’re protecting – you can’t protect what you don’t know.
"The first part to combat data poisoning in AI-driven SaaS products is to have a clear understanding of what data you are training your model on,” says Ian Ahl, SVP of P0 Labs, the threat research arm of identity security provider Permiso. “It's critical to train models on data you can control.”
What kind of data is it? Where is it coming from? Who has access to it? If you don’t have clear answers to questions like these, you’re probably at higher risk of a poisoning attack.
A strong understanding of your training data is also a prerequisite for good data governance, which we’ll unpack below. But suffice it to say for now, there need to be controls in place for how people, teams, and processes interact with and use training data.
“By evaluating the data itself, you can put tighter controls on the users who are updating it and have strict guardrails around how customers can interact with that data,” Ahl says. “It's a blend of oversight on people, processes, and data."
2. Revisit your data transformation strategy.
You also need to consider – and potentially reconsider – how your data gets from its original sources to your models, also known as a data pipeline.
“The decisions CTOs are making about data transformation strategy directly impact the security of their AI-powered SaaS products because those decisions affect just how vulnerable data pipelines can be to poisoning,” says Dave Jenkins, VP of Product and Research at Iterate.ai.
Specifically, Jenkins says we’re in the midst of a broader shift from ETL (Extract, Transform, Load) to ELT (Extract, Load, Transform) data pipeline architecture. There are multiple reasons for the trend, but one advantage of the latter strategy is that it is better-suited to minimize the risk of problems such as data poisoning.
“The reasoning behind this is that when AI data transforms happen early, and the model is poorly tuned or data sets are still incomplete, as happens sometimes with ETL, that potentially bad data gets baked in,” Jenkins says. “Errors and hallucinations can multiply from there, and finding – and correcting – the source of the poisoned data can be all but impossible.”
Because of that, Jenkins adds, you want to avoid a scenario where different applications do their own transformations before the data gets moved into a data lake. The ELT model flips the order of operations, so to speak, and gives you more control and a more unified transformation process.
“By instead moving all transformations into a data warehouse environment like Snowflake, CTOs and their teams gain more visibility and control as they build out their products,” Jenkins says.
3. Prioritize good data governance.
Visibility and control are critical, but they’re incomplete without strong policies and governance throughout your data pipelines and processes.
“Getting this right, of course, also requires clear governance,” Jenkins says. “CTOs must specify where AI data transformations can occur. They'll want to have audit trails, validation frameworks, reference datasets for comparison testing, and, ideally, transformation sandboxes to test and retest for any errors or hallucinations before production.”
Ahl notes that governance needs to include people, too, particularly in the form of clear policies and guardrails around who can interact with training data, what data they can use, and so forth: “Whoever has the ability to add files to train a model, whether that is a user or a team, there need to be constraints on the data they can leverage.”
4. Shore up your overall security posture.
It’s also important to remember that AI in general is yet again expanding the attack surface of many organizations. Data poisoning is just one potential threat – albeit an important one. All of the above should be happening in the context of a broader security strategy.
If you already punt security fundamentals like patching, password hygiene, identity management, and the principle of least privilege, then it follows that your AI models and training data will be even more susceptible to risks, too.
5. Consider adversarial model training.
When newer threats like data poisoning emerge, new strategies and tools for defending against those threats usually follow. That’s happening now with AI, which has arrived with a whole bundle of new risks, not just data poisoning. These are sometimes grouped under the term “adversarial AI,” or the malicious use of machine learning and other types of AI to attack other AI systems. Data poisoning is one form of adversarial AI.
You can flip the script here and use adversarial training, which essentially teaches your models how to spot potential patterns and anomalies that might indicate data poisoning other forms of adversarial AI and then take steps to mitigate them.
Adversarial training continues to gain steam as a key approach to AI security. For example, this academic paper offers a framework for mitigating biases that may have been acquired through data collection in healthcare settings, where the potential consequences of data poisoning and other attacks are especially serious. Another paper proposes an adversarial training framework specifically for defending against data poisoning.
Need expert help selecting the right tool?
With one-on-one help, we guide you to your top software options. Narrow down your software search & make a confident choice.
What's Next?
If you’re building generative AI and other AI-fueled capabilities into your SaaS product portfolio, you must ensure your training data is clean, accurate, and secure.
Otherwise, you increase your risks of data poisoning and other problems – and those problems are usually easier to prevent than to resolve after the fact.
Before you go, sign up for our newsletter to get the latest insights!
