Inference Speed: 31K samples/second

Learning Objectives

By the end of this section, you will:

1. Understand AMNL's inference throughput of 31K samples/second
2. Analyze latency characteristics for real-time deployment
3. Identify computational bottlenecks in the inference pipeline
4. Learn optimization strategies for production deployment
5. Compare with industrial requirements for predictive maintenance

Core Insight: AMNL processes 31,625 samples per second on a single GPU—equivalent to monitoring over 31,000 engines simultaneously with 1-second update intervals. This throughput far exceeds typical industrial requirements, enabling real-time monitoring at scale.

Inference Throughput

Inference throughput measures how many predictions the model can make per unit time. This is critical for industrial deployment where thousands of assets may need simultaneous monitoring.

Benchmark Results

Throughput by Batch Size

Throughput varies significantly with batch size due to GPU utilization efficiency:

What Does 31K Samples/Second Mean?

Latency Analysis

While throughput measures overall capacity, latency measures how quickly an individual prediction is returned—critical for real-time alerting systems.

Latency Breakdown

Latency Distribution

Latency varies across samples due to GPU scheduling and memory access patterns:

Computational Bottlenecks

Understanding where computation time is spent helps identify optimization opportunities.

BiLSTM Dominates

The BiLSTM encoder accounts for 39.2% of inference time, making it the primary bottleneck:

• Sequential nature: LSTMs process timesteps sequentially, limiting parallelization
• 3-layer depth: Each layer adds latency
• Bidirectional processing: Effectively 6 LSTM passes

Attention Overhead

Multi-head attention contributes 19.3% of latency with 12 attention heads:

Where $n$ is sequence length, $d$ is embedding dimension, and $h$ is number of heads.

Memory Bandwidth

Optimization Strategies

Several strategies can further improve inference speed for production deployment.

1. Mixed Precision Inference

1import torch
2
3# Enable automatic mixed precision for inference
4model = model.half()  # Convert to FP16
5
6with torch.cuda.amp.autocast():
7    predictions = model(input_batch)
8
9# Speedup: ~1.5-2x on modern GPUs

2. ONNX Runtime Optimization

1import onnxruntime as ort
2
3# Export to ONNX
4torch.onnx.export(model, dummy_input, "amnl.onnx")
5
6# Load with optimizations
7sess_options = ort.SessionOptions()
8sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
9sess = ort.InferenceSession("amnl.onnx", sess_options)
10
11# Speedup: ~1.3x over PyTorch

3. TensorRT Compilation

1import torch_tensorrt
2
3# Compile for TensorRT
4trt_model = torch_tensorrt.compile(
5    model,
6    inputs=[torch_tensorrt.Input((256, 50, 17))],
7    enabled_precisions={torch.float16}
8)
9
10# Speedup: ~2-3x over PyTorch FP32

Optimization Summary

Summary

Inference Speed Analysis - Summary:

1. Base throughput: 31,625 samples/second on RTX 5000
2. Per-sample latency: 31.6 μs average, 48.7 μs at p99
3. Primary bottleneck: BiLSTM encoder (39.2% of time)
4. Easy optimization: FP16 provides 1.5× speedup with no code changes
5. Maximum throughput: ~95K samples/sec with TensorRT INT8

Key Insight: AMNL's inference speed of 31K samples/second exceeds industrial requirements by an order of magnitude for typical deployments. This headroom enables real-time monitoring at scale, high-frequency update rates, and room for future growth—all on a single GPU without specialized optimization. For extreme-scale deployments, TensorRT optimization can push throughput to nearly 100K samples/second.