Introduction
Traditional security risk assessments focus on software vulnerabilities, network exposures, and access control weaknesses. AI systems introduce entirely new risk categories that existing frameworks are not equipped to evaluate: model drift, training data quality degradation, adversarial robustness, and emergent behaviors that were not present during testing.
A comprehensive AI security risk assessment must extend traditional threat modeling methodologies to cover the unique properties of machine learning systems. This requires collaboration between security engineers, data scientists, and governance teams to identify, evaluate, and mitigate risks across the entire AI lifecycle.
Threat Modeling for AI Systems
STRIDE, the classic threat modeling framework (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege), can be extended for ML systems by mapping each category to AI-specific threats. Tampering, for instance, encompasses not just data modification but training data poisoning and model weight manipulation.
An effective AI threat model must consider the full ML pipeline as an attack surface: data collection and labeling, feature engineering, model training, validation, deployment, and inference. Each stage presents distinct threat vectors and requires tailored security controls.
- Spoofing: Adversarial examples that impersonate legitimate inputs to bypass AI-based authentication or detection systems
- Tampering: Data poisoning, model backdoors, and manipulation of training pipelines to alter model behavior
- Information Disclosure: Model extraction, membership inference, and training data leakage through model outputs
- Denial of Service: Inputs designed to cause excessive computation, model crashes, or resource exhaustion
- Elevation of Privilege: Prompt injection attacks that cause AI agents to perform unauthorized actions
The threat model should also consider supply chain risks specific to AI: compromised pre-trained models, poisoned datasets from third-party providers, and vulnerabilities in ML frameworks and libraries. Each external dependency is a potential entry point for adversarial influence.
Key Insight: Traditional threat modeling assumes deterministic software behavior. AI threat models must account for the probabilistic nature of ML systems, where the same input can produce different outputs and where model behavior changes over time as data distributions shift.
AI-Specific Risk Categories
Beyond the adversarial threats covered in previous chapters, AI systems face operational risks that can degrade security effectiveness without any attacker involvement. These risks must be identified, measured, and managed as part of a comprehensive risk assessment program.
Model drift is perhaps the most insidious operational risk. As the real-world data distribution evolves—new attack techniques emerge, network traffic patterns change, user behaviors shift—a model trained on historical data becomes increasingly misaligned with current conditions. Without monitoring, a model that was highly effective at deployment can silently degrade to the point of uselessness.
- Model drift: Gradual degradation of model performance as the real-world data distribution diverges from the training distribution
- Training data quality: Errors, biases, or staleness in training data that compromise model reliability from the outset
- Concept drift: Changes in the underlying relationships between features and labels that invalidate the patterns the model learned
- Feedback loops: Model predictions influencing the data used for retraining, creating self-reinforcing biases
- Emergent behaviors: Unexpected model behaviors that arise in production contexts not represented in testing environments
Each of these risk categories requires specific monitoring strategies and mitigation plans. Model drift detection, for example, can be implemented through statistical tests comparing incoming data distributions to the training distribution, with automated alerts when significant divergence is detected.
Continuous Monitoring and AI Auditing
AI security is not a point-in-time assessment but a continuous process. Models change behavior over time, new attack techniques emerge, and the data landscape evolves. Continuous monitoring provides the visibility needed to detect degradation, attacks, and compliance violations before they cause significant harm.
AI auditing complements monitoring by providing structured, periodic evaluations of AI system compliance, performance, and security posture. Audits should cover the entire ML lifecycle, from data sourcing and training practices to deployment configurations and operational metrics.
An effective monitoring and auditing program tracks key metrics including model accuracy on held-out test sets, prediction confidence distributions, feature importance stability, data pipeline integrity, and adversarial robustness scores. Deviations from baseline values trigger investigation and remediation workflows.
Why This Matters: Regulations like the EU AI Act mandate post-market monitoring for high-risk AI systems. Organizations that implement robust continuous monitoring and auditing programs will not only meet compliance requirements but will also detect and respond to security issues faster, reducing the window of exposure when vulnerabilities are exploited.