Network Intrusion Detection with ML

Introduction

Building a machine learning-powered Network Intrusion Detection System (NIDS) requires careful consideration of data sources, feature engineering, and model selection. Unlike traditional classification tasks, network intrusion detection operates in an adversarial environment where attackers actively try to evade detection, and the class distribution is heavily imbalanced—malicious traffic typically represents less than 1% of all network flows.

This section walks through the complete ML pipeline for network intrusion detection, from selecting appropriate datasets to training and evaluating models that can operate at production scale.

Benchmark Datasets for Network IDS

The quality of any ML-IDS depends heavily on the training data. The security community has produced several benchmark datasets specifically designed for intrusion detection research. Understanding their strengths and limitations is critical for building robust models.

The CICIDS2017 dataset, created by the Canadian Institute for Cybersecurity, contains benign traffic and the most common attack types including DDoS, brute force, web attacks, and infiltration. It provides over 80 extracted network flow features and labeled attack categories, making it one of the most widely used benchmarks in academic research.

The UNSW-NB15 dataset offers a more modern alternative with 49 features across nine attack families. It addresses several criticisms of older datasets like KDD Cup 99, which suffered from redundant records and unrealistic traffic distributions. For production systems, combining multiple datasets and supplementing with organization-specific data is essential.

• CICIDS2017: 80+ features, 14 attack types, realistic traffic patterns from 5 days of capture
• UNSW-NB15: 49 features, 9 attack families, addresses KDD99 limitations
• CSE-CIC-IDS2018: Updated version with 80+ features and 7 attack scenarios
• CTU-13: 13 botnet scenarios with real botnet traffic captures

Feature Extraction from Network Traffic

Raw network packets must be transformed into meaningful features before ML models can process them. This feature engineering step is arguably the most important part of the ML-IDS pipeline, as the quality of features directly determines detection accuracy.

Flow-level features aggregate information from related packets into a single record describing a network conversation. These include duration, total bytes transferred, packet counts, inter-arrival times, and protocol-specific fields. Statistical features such as mean, variance, and entropy of packet sizes within a flow capture behavioral patterns that distinguish normal from malicious traffic.

1# Example: Extracting flow features with CICFlowMeter
2# Key features for ML-IDS models
3flow_features = {
4    "duration": "Total flow duration in microseconds",
5    "total_fwd_packets": "Packets sent in forward direction",
6    "total_bwd_packets": "Packets sent in backward direction",
7    "flow_bytes_per_sec": "Flow throughput in bytes/second",
8    "flow_iat_mean": "Mean inter-arrival time between packets",
9    "fwd_psh_flags": "Number of PSH flags in forward direction",
10    "bwd_packet_length_std": "Std dev of backward packet lengths",
11    "flow_entropy": "Shannon entropy of payload bytes",
12}

Supervised Models: Random Forest, XGBoost, and Beyond

For labeled datasets, supervised learning models offer the highest detection accuracy. Tree-based ensemble methods—Random Forest and XGBoost in particular—have consistently demonstrated superior performance on network intrusion detection tasks, often achieving F1 scores above 0.99 on benchmark datasets.

Random Forest builds multiple decision trees on random subsets of features and data, combining their predictions through majority voting. This ensemble approach provides natural resistance to overfitting and handles the high-dimensional feature spaces common in network data. XGBoost adds gradient boosting to the ensemble, iteratively correcting errors from previous trees.

However, benchmark performance does not guarantee production effectiveness. Models trained on CICIDS2017 may not generalize to enterprise networks with different traffic patterns, application mixes, and attack profiles. Cross-domain evaluation and regular retraining are essential for maintaining detection accuracy in deployment.

Important Caveat: High accuracy on benchmark datasets can be misleading. A model that achieves 99.9% accuracy may still generate thousands of false positives daily on a high-throughput enterprise network processing millions of flows per hour. Always evaluate models using precision, recall, and F1 score rather than accuracy alone.

Unsupervised Anomaly Detection

When labeled data is unavailable or incomplete, unsupervised anomaly detection methods learn what "normal" network behavior looks like and flag deviations. Isolation Forest is particularly effective for network data, as it identifies anomalies by measuring how easily a data point can be separated from the rest of the dataset.

Autoencoders offer another powerful approach. These neural networks are trained to compress and reconstruct normal network traffic. When presented with anomalous traffic, the reconstruction error increases significantly, providing a natural anomaly score. Deep autoencoders can capture complex nonlinear relationships between features that linear methods miss.

1. Isolation Forest: Partitions data randomly; anomalies require fewer partitions to isolate
2. Autoencoders: Learn compressed representation of normal traffic; high reconstruction error signals anomalies
3. One-Class SVM: Learns a boundary around normal data in high-dimensional feature space
4. DBSCAN: Density-based clustering that identifies outliers as points in low-density regions

Online Learning for Evolving Threats

Network traffic patterns evolve continuously as applications change, user behavior shifts, and new services are deployed. Static models trained on historical data degrade over time, a phenomenon known as concept drift. Online learning algorithms address this by updating the model incrementally as new data arrives.

Hoeffding trees and online random forests can incorporate new training examples without retraining from scratch, making them suitable for streaming network data. Adaptive windowing techniques like ADWIN automatically detect when the underlying data distribution has changed and adjust the model accordingly.

The challenge with online learning in an adversarial context is ensuring that attackers cannot manipulate the learning process itself. If an attacker can gradually shift what the model considers "normal," they can train the IDS to ignore their malicious traffic—a form of data poisoning specific to online learning systems.