Introduction
Not every AI technique is equally useful for cybersecurity. While the field of machine learning encompasses hundreds of algorithms and architectures, security applications consistently rely on a core set of tools. This section identifies the AI/ML techniques that matter most for security engineers, explains when and why each is used, and provides the conceptual foundation you need before building real detection systems.
Whether you are classifying malware, detecting network intrusions, or analyzing threat intelligence reports, the techniques in this section will form your daily toolkit. We focus on intuition and application rather than mathematical derivation, though we point you to deeper resources where appropriate.
Learning Paradigms for Security
Machine learning is broadly divided into three paradigms, each with distinct applications in cybersecurity. Understanding which paradigm fits a given security problem is the first step in designing an effective solution.
Supervised learning trains models on labeled data—examples where the correct answer is known. In security, this means training on datasets where traffic is labeled as benign or malicious, files are labeled as clean or infected, and emails are labeled as legitimate or phishing. Supervised learning excels when labeled data is available but struggles with novel attacks not represented in the training set.
Unsupervised learning finds patterns in unlabeled data. For security, this means detecting anomalies—deviations from normal behavior—without needing examples of specific attacks. This is particularly valuable for zero-day detection, insider threat identification, and discovering unknown attack patterns.
- Supervised Learning: Malware classification, phishing detection, spam filtering, vulnerability severity prediction. Requires labeled datasets.
- Unsupervised Learning: Anomaly detection, user behavior analytics, network traffic clustering, unknown threat discovery. Works without labels.
- Reinforcement Learning (RL): Automated penetration testing, adaptive defense strategies, security policy optimization. The agent learns through trial and error in simulated environments.
Practical Guidance: Start with supervised learning when you have labeled data—it produces the most reliable results. Use unsupervised methods for discovery and anomaly detection where labels are unavailable. Reserve RL for specialized applications like autonomous security agents, which we cover in Chapter 22.
Key Classical Algorithms
Despite the popularity of deep learning, classical ML algorithms remain workhorses in production security systems. They train faster, require less data, are more interpretable, and often perform comparably to deep learning on tabular security data (logs, flow records, feature vectors).
Decision Trees split data based on feature thresholds, creating interpretable if-then rules. A single decision tree is easy to understand but prone to overfitting. Random Forests address this by training many trees on random subsets of data and features, then voting on the final prediction. Random forests are the go-to baseline for most security classification tasks.
- Decision Trees: Highly interpretable, good for creating security rules from data. Limited accuracy on complex problems.
- Random Forests: Robust, accurate, handles missing data well. Excellent baseline for intrusion detection and malware classification.
- Support Vector Machines (SVMs): Effective for binary classification with clear margins. Used in malware detection and anomaly detection with kernel tricks.
- k-Means Clustering: Groups similar data points together. Used for network traffic profiling, alert clustering, and threat group analysis.
- Gradient Boosted Trees (XGBoost, LightGBM): State-of-the-art for tabular data. Wins most security ML competitions and powers production detection systems.
Deep Learning Architectures
Deep learning models excel when security data has spatial structure (binary files, network packet payloads), sequential structure (log sequences, command histories), or when the feature space is too complex for manual engineering.
Convolutional Neural Networks (CNNs) apply learnable filters to detect local patterns. In security, CNNs are used to classify malware by treating binary files as grayscale images, detect malicious traffic patterns in packet payloads, and identify visual similarities between phishing pages and legitimate sites.
Recurrent Neural Networks (RNNs) and their variants (LSTM, GRU) process sequential data by maintaining hidden state across time steps. Security applications include analyzing sequences of system calls for behavioral malware detection, modeling user session patterns for anomaly detection, and processing log streams for real-time threat detection.
- CNNs: Malware visualization, network traffic analysis, image-based phishing detection
- RNNs/LSTMs: System call analysis, log sequence modeling, behavioral anomaly detection
- Transformers: NLP for threat intelligence, code analysis, log understanding, and LLM-powered security tools
- Autoencoders: Anomaly detection via reconstruction error, dimensionality reduction for security features
- GANs: Generating synthetic attack data for training, adversarial robustness testing
When to Use Deep Learning: If your security data is tabular (logs, flow records with extracted features), start with gradient boosted trees. Switch to deep learning when working with raw bytes, images, natural language, or sequences where manual feature engineering is insufficient. Deep learning requires more data and compute but can discover patterns that classical methods miss.