Introduction
Machine learning models are not merely software—they are complex mathematical functions trained on vast datasets, and every stage of their lifecycle presents opportunities for attack. From the training data they consume to the predictions they serve, AI systems carry a vulnerability surface that is fundamentally different from traditional software.
Understanding this vulnerability surface is the first step toward building robust, production-ready AI systems. In this section, we map the attack landscape and examine why adversarial machine learning has become one of the most critical fields in modern cybersecurity.
AI Models as Attack Targets
Traditional software vulnerabilities—buffer overflows, SQL injection, cross-site scripting—target deterministic code paths. AI models, by contrast, are probabilistic systems whose behavior is defined not by explicit rules but by learned patterns in data. This makes them susceptible to entirely new classes of attacks that exploit the statistical nature of machine learning.
An attacker does not need to find a bug in the source code. Instead, they can manipulate inputs, corrupt training data, or reverse-engineer model internals—all without ever touching the underlying infrastructure. The model itself becomes the vulnerability.
This paradigm shift means that security engineers must think beyond perimeter defenses and consider the model as an attack surface in its own right. Every API endpoint serving predictions, every training pipeline ingesting data, and every model artifact stored on disk is a potential entry point.
Key Insight: Unlike traditional software bugs, adversarial ML vulnerabilities are inherent to how models learn. You cannot "patch" a model's susceptibility to adversarial examples the way you patch a buffer overflow—it requires fundamentally rethinking how models are trained and deployed.
Taxonomy of Adversarial Attacks
Adversarial attacks against ML systems fall into four broad categories, each targeting a different phase of the model lifecycle. Understanding this taxonomy is essential for prioritizing defenses and assessing risk.
- Evasion Attacks: Crafting inputs at inference time that cause the model to misclassify. The attacker modifies test-time inputs with imperceptible perturbations to fool a deployed model.
- Poisoning Attacks: Corrupting the training data so the model learns incorrect patterns. This can introduce backdoors that activate only on specific trigger inputs.
- Model Extraction: Querying a deployed model repeatedly to reconstruct a functionally equivalent copy, stealing proprietary intellectual property and enabling further attacks.
- Model Inversion: Exploiting model outputs to recover sensitive information about the training data, violating privacy guarantees and potentially exposing personal records.
Each category demands distinct defensive strategies, and a comprehensive security posture must address all four. In practice, attackers often combine techniques—for example, extracting a model first and then using the copy to develop evasion attacks offline.
The 2025 Shift: GenAI Data Leaks
The emergence of generative AI has reshaped the adversarial ML threat landscape dramatically. By 2025, industry surveys show that 34% of organizations cite GenAI data leakage as their top AI security concern, surpassing even the 29% who worry about traditional adversarial attacks on classification models.
This shift reflects a fundamental change in how AI is deployed. When models were primarily classifiers operating on structured data, the risk was misclassification. Now that large language models process and generate natural language, the risk extends to inadvertent disclosure of training data, system prompts, and confidential information fed through retrieval-augmented generation pipelines.
Why This Matters: The adversarial ML field is no longer limited to academic research on image perturbations. It now encompasses the full spectrum of generative AI risks, from prompt injection to training data extraction, making it a boardroom-level concern for every organization deploying AI.
Security teams must expand their threat models to account for these new attack vectors. The chapters that follow will explore each category of adversarial attack in depth, providing both the theoretical foundation and practical defenses needed to secure modern AI systems.