Data Contamination Detection

We have spent six sections building a pipeline that turns raw internet text into a clean, mixed, scheduled stream of 14.8T training tokens. One question remains, and it is the one that decides whether the model you ship is honest. How do you make sure your model has never seen the test?

The thesis of this section. Benchmark scores are the public-facing measure of a frontier model's quality. If a single copy of an MMLU question, a HumanEval prompt, or a GSM8K problem ends up in the pretraining corpus, the model can memorise the answer verbatim and post inflated scores that mean nothing. Contamination detection is the audit that keeps the leaderboard honest — and at 14.8T-token scale it is a non-trivial systems problem.

The Real Problem: A Test the Model Already Saw

Imagine you spend $5M of GPU time training a 671B-parameter model. You evaluate it on MMLU, GSM8K, HumanEval, MATH, BBH, and a dozen other public benchmarks. The scores look great — better than any previous open model. You ship. Within a week, an independent researcher discovers that one of the benchmark questions appeared verbatim in a Stack Exchange post that was crawled by Common Crawl in 2023. With an effective epoch count of ~8 on that document (because it was upweighted as a high-quality math source — see section 8.4), the model saw the question and its answer eight times. The benchmark score is meaningless. Your release is dead.

This is not a hypothetical. Three real cases:

The pattern is consistent. Models that win leaderboards by margins that look too good to be true usually are too good to be true — and the explanation, more often than architecture or scale, is that a few thousand benchmark items leaked through the data pipeline. The contamination scanner is the last line of defence between an honest evaluation and an embarrassing retraction.

Intuition: Catching Plagiarism at Web Scale

The mental model is academic plagiarism detection — but inverted. Plagiarism software checks whether a student paper overlaps with a known reference corpus. Contamination detection checks whether a known eval set (the "reference") overlaps with a candidate training corpus (the "student"). Same math, opposite roles.

Three constraints make this harder than classroom plagiarism:

1. Scale. A plagiarism checker handles a few thousand papers against a few million references. We handle trillions of training tokens against tens of thousands of eval questions. The naive all-pairs comparison is $O(N \cdot M)$ — completely impossible at 14.8T × 50k.
2. Paraphrase tolerance. The leak is rarely verbatim. A Stack Overflow post will rephrase the eval prompt, drop a word, add a comment. We need to catch "the same question with one word changed" — but we must not catch "a totally different question about the same topic." This is the fundamental tension n-gram length controls.
3. Streaming. We cannot load the corpus into memory. We get one document at a time, must make a keep/drop decision in milliseconds, and move on. The eval index lives in RAM; the training corpus streams past.

The dominant technique in 2020–2025 — used by GPT-3, Llama-3, DeepSeek-V3 — is n-gram overlap detection: chop both sides into fixed-length windows of $n$ tokens and count how many of the eval's windows appear in the training document. Simple, fast, and surprisingly hard to defeat.

The Mathematics of N-Gram Overlap

Let $d_e$ be an eval document and $d_t$ a candidate training document. After whitespace tokenisation we have two token sequences. Define the set of n-grams of a sequence $s = (s_1, \dots, s_L)$ as:

$G_n(s) = \{ (s_i, s_{i+1}, \dots, s_{i+n-1}) : 1 \leq i \leq L - n + 1 \}$

The contamination ratio of $d_t$ with respect to $d_e$ is the fraction of eval n-grams that also appear in the training document:

$c_n(d_e, d_t) = \frac{|G_n(d_e) \cap G_n(d_t)|}{|G_n(d_e)|}$

This is a number in $[0, 1]$. $c_n = 0$ means no overlap; $c_n = 1$ means every n-gram of the eval doc also appears in the training doc — total verbatim leak. The decision rule is a pair of thresholds:

$\text{verdict}(d_t) = \begin{cases} \text{DROP} & \text{if } \max_{e} c_n(d_e, d_t) \geq \tau_{\text{drop}} \\ \text{FLAG} & \text{if } \tau_{\text{flag}} \leq \max_e c_n(d_e, d_t) < \tau_{\text{drop}} \\ \text{KEEP} & \text{otherwise} \end{cases}$

The standard choices in published frontier models are $n = 13, \; \tau_{\text{flag}} = 0.10, \; \tau_{\text{drop}} = 0.50$. The $\max_e$ over eval documents is critical — a document that leaks 80% of one benchmark and 0% of others must still drop.

Why the value of n matters

Two probabilities compete inside the choice of $n$. Let $p$ be the probability that a single token matches by chance — about $1/V$ for vocabulary size $V \approx 50{,}000$. The probability that an entire n-gram collides by chance is roughly $p^n$ — astronomically small for $n = 13$. So a 13-gram match is almost never coincidence; it is a real leak. But a paraphrase that swaps one word out of every 13 will still trigger a match on the surrounding 12 windows — so the filter remains paraphrase-tolerant up to the rate of about one substitution per $n$ tokens.

Drop $n$ to 8 and the chance of a coincidental collision rises (false positives), but the filter catches heavier paraphrases. Raise $n$ to 25 and you catch only verbatim copies (false negatives explode). The sweet spot at 13 was found empirically by the GPT-3 team and has stuck.

Why we use Jaccard-style ratio, not edit distance

Edit distance and embedding similarity would give a more semantically-aware measure of leakage. They are also $O(L^2)$ or worse per pairwise comparison — completely intractable at 14.8T tokens. N-gram set intersection is $O(L)$ with hash-set lookup. Frontier labs run a fast n-gram pre-filter to drop the obvious 99%, then an expensive embedding-based scan over the remaining 1% for the borderline cases. We focus on the n-gram pre-filter because it does the heavy lifting.

Manual Numerical Walkthrough

Let us compute the contamination ratio by hand for two real-looking documents at $n = 5$ (small enough to do on paper, large enough to be meaningful).

Visualizing Contamination

The scanner below pairs three real-looking eval/training pairs against each other. Drag the n-gram slider to watch the contamination ratio change: short n catches paraphrases at the cost of false positives, long n catches only verbatim leaks. The badge at the top shows the production verdict — green KEEP, amber FLAG, red DROP — at the current thresholds.

Three things to internalise from the sandbox. First, on the clean example the contamination ratio is exactly zero for any $n \geq 4$ — clean training data does not accidentally collide with the benchmark, no matter the n. Second, on the paraphrase example the ratio decays sharply with $n$: short n flags it loudly, long n misses it entirely. Third, on the verbatim leak the ratio sits near 1.0 for every $n$ the document is long enough to support — the leak is unmistakable. The engineering question is where to draw the line so paraphrases are flagged and topical-similar docs are kept.

Plain Python: An N-Gram Contamination Scanner

Below is the complete contamination scanner in plain Python — no external libraries, no GPUs. Three benchmark items, three candidate training docs, and a per-document verdict. This is exactly the algorithm GPT-3 published in 2020 (Brown et al., section 5).

1from collections import defaultdict
2
3# The benchmark set we want to protect — the questions the model
4# will be GRADED on later. We must NEVER let the model see these
5# (or close paraphrases) during pretraining.
6EVAL = [
7    "What is the capital of Australia? The capital of Australia is Canberra.",
8    "Write a Python function that returns the sum of all even numbers in a list.",
9    "Natalia sold clips to 48 of her friends in April and then sold half as many in May.",
10]
11
12# A small batch of candidate training documents.
13TRAIN = [
14    "Canberra, the capital of Australia, was purpose-built between 1913 and 1927 ...",
15    "Solution: write a Python function that returns the sum of all even numbers in a list. "
16    "The cleanest version is sum(x for x in nums if x % 2 == 0).",
17    "When teaching ratios, ask students to draw a bar model for each month.",
18]
19
20N = 13                # n-gram length used by GPT-3, Llama-3, DeepSeek
21FLAG_THRESHOLD = 0.10 # >=10% overlap => human review
22DROP_THRESHOLD = 0.50 # >=50% overlap => discard document
23
24def tokenize(s):
25    return s.lower().replace(",", " ").replace(".", " ").replace("?", " ").split()
26
27def ngrams(toks, n):
28    return [tuple(toks[i:i+n]) for i in range(len(toks) - n + 1)]
29
30# 1) Build the EVAL n-gram index ONCE, up front.
31#    Key insight: this is the small side of the problem (MB),
32#    the training corpus is the big side (TB).
33eval_grams = defaultdict(set)   # gram -> {eval_doc_ids that contain it}
34for doc_id, text in enumerate(EVAL):
35    for g in ngrams(tokenize(text), N):
36        eval_grams[g].add(doc_id)
37
38# 2) Stream over training docs. For each doc, count how many of
39#    its n-grams collide with the index, then bucket per eval doc.
40def scan(train_text):
41    grams = ngrams(tokenize(train_text), N)
42    if not grams:
43        return 0.0, {}
44    hits_per_eval = defaultdict(int)
45    total_eval_grams = {i: 0 for i in range(len(EVAL))}
46    for i, e in enumerate(EVAL):
47        total_eval_grams[i] = max(1, len(ngrams(tokenize(e), N)))
48    matched = 0
49    for g in grams:
50        if g in eval_grams:
51            matched += 1
52            for eid in eval_grams[g]:
53                hits_per_eval[eid] += 1
54    # Per-eval contamination = matched grams shared with eval_i / |eval_i grams|
55    ratios = {eid: hits_per_eval[eid] / total_eval_grams[eid]
56              for eid in hits_per_eval}
57    overall = matched / len(grams)
58    return overall, ratios
59
60# 3) Decide: drop, flag, or keep.
61for tid, text in enumerate(TRAIN):
62    overall, ratios = scan(text)
63    worst_eval = max(ratios.values(), default=0.0)
64    if worst_eval >= DROP_THRESHOLD:
65        verdict = "DROP"
66    elif worst_eval >= FLAG_THRESHOLD:
67        verdict = "FLAG"
68    else:
69        verdict = "KEEP"
70    print(f"train[{tid}] verdict={verdict:4s} worst_eval_overlap={worst_eval:.3f}")

Two structural details worth a second look. First, the index is built on the eval set, not on the training set. This is the difference between a 30-second job and a multi-day job. The eval set is the small side and never changes; index it once, query it billions of times. Second, the per-eval breakdown (lines 53–58) lets downstream tooling answer the question "which benchmark did this document leak?" — without it, the contamination report is just a list of dropped documents with no actionable signal.

Sanity check yourself. Run this scanner with EVAL = TRAIN (set the eval and training corpora to the same list). Every document should report a contamination ratio of exactly 1.0 against its matching eval doc. If yours come back at 0.0, the n-gram extraction is broken — almost always an off-by-one in the sliding window.

PyTorch: Bloom Filters and Sharded Scans at Scale

At 14.8T tokens the dict-of-grams approach hits two walls: memory (the eval gram set grows when you index more benchmarks) and throughput (every gram lookup is a Python hash + string compare). The production version swaps the dict for a Bloom filter and wraps the scan in a sharded IterableDataset that runs across hundreds of CPU workers.

1import hashlib
2import torch
3from torch.utils.data import IterableDataset, DataLoader
4
5# At 14.8T tokens, the eval n-gram set is too big to keep in a Python
6# dict — we move to a Bloom filter, which is a tiny probabilistic set.
7class BloomFilter:
8    def __init__(self, n_bits: int, n_hashes: int):
9        # n_bits ~= 2 ** 34 for a few hundred thousand eval grams
10        # with false-positive rate < 1e-4
11        self.bits = torch.zeros(n_bits, dtype=torch.bool)
12        self.n_bits = n_bits
13        self.n_hashes = n_hashes
14
15    def _hashes(self, gram: bytes):
16        # k independent hashes from one SHA-256
17        h = hashlib.sha256(gram).digest()
18        return [int.from_bytes(h[i*4:(i+1)*4], "big") % self.n_bits
19                for i in range(self.n_hashes)]
20
21    def add(self, gram: bytes):
22        for h in self._hashes(gram):
23            self.bits[h] = True
24
25    def __contains__(self, gram: bytes) -> bool:
26        return all(self.bits[h] for h in self._hashes(gram))
27
28# 1) Build the eval Bloom filter ONCE on a head node.
29eval_bloom = BloomFilter(n_bits=2**34, n_hashes=7)
30for doc in load_eval_corpus():            # ~ a few MB
31    for gram in document_ngrams(doc, n=13):
32        eval_bloom.add(gram)
33
34# 2) Each worker streams a shard of the training corpus.
35class ContaminationFilter(IterableDataset):
36    def __init__(self, shard_paths, bloom, n=13, drop_thresh=0.5):
37        self.shard_paths = shard_paths
38        self.bloom = bloom
39        self.n = n
40        self.drop_thresh = drop_thresh
41
42    def __iter__(self):
43        for path in self.shard_paths:
44            for doc in stream_documents(path):       # one doc at a time
45                grams = list(document_ngrams(doc, self.n))
46                if not grams:
47                    continue
48                hits = sum(1 for g in grams if g in self.bloom)
49                ratio = hits / len(grams)
50                if ratio >= self.drop_thresh:
51                    log_dropped(doc, ratio)          # forensic trail
52                    continue
53                yield doc                            # keep this doc
54
55# 3) Wire into the standard DataLoader so contamination filtering
56#    happens at corpus-build time, NOT at training time.
57shards = ["/data/web/0.bin", "/data/web/1.bin", ...]
58clean_stream = ContaminationFilter(shards, eval_bloom)
59loader = DataLoader(clean_stream, batch_size=1, num_workers=128)
60for clean_doc in loader:
61    write_to_clean_shard(clean_doc)

Three subtleties worth marking, all about how this filter interacts with the rest of the data pipeline:

1. The Bloom filter is read-only after construction. We build it on a head node, write it to a shared blob, and every worker memory-maps the same bitset. There is no synchronisation, no consistency problem — just a giant read-only array. This is the single design choice that turns the scanner from "parallelisable in principle" to "embarrassingly parallel."
2. Filtering is corpus-build-time, not training-time. The DataLoader you see above runs ONCE, before training. Its output is a stream of clean documents written to a new set of shards. The training loop later reads only from those clean shards — no contamination logic anywhere near the gradient updates. Mixing the two is the most common architectural mistake.
3. The drop log is the audit trail. log_dropped(doc, ratio) writes (doc-hash, contamination-ratio, top-matching-eval-id) to a side-channel log. DeepSeek-V3's release report includes this log as a public appendix — anyone can verify which documents were filtered and why. The log is also the way you re-run the scan with a tighter threshold without re-processing the corpus: just re-filter the log itself.

At Massive Scale: 14.8T Tokens vs Dozens of Benchmarks

Plug the production numbers into the algorithm and the systems story becomes concrete. DeepSeek-V3 pretrains on 14.8T tokens. After BPE tokenisation that is roughly the same number of n-gram windows. The eval suite is ~50k items pulled from ~30 public benchmarks, which produces roughly 1–2 million distinct 13-grams. The full scan is:

Two systems-level observations. First, the contamination scan is sub-1% of the total training cost — there is no economic reason ever to skip it. Second, the ~0.03% drop rate sounds tiny until you remember the contamination budget asymmetry: dropping clean documents costs you a fraction of a bit of validation loss; keeping a leaked document inflates a benchmark score by 5–30 percentage points and burns your release credibility. The expected value math is overwhelming in favour of aggressive filtering.

Model-based contamination detection

N-gram filtering catches surface-level overlap. The 2024 generation of frontier models added a second, complementary technique: model-based contamination detection. Run the trained model over the eval question with the answer text masked, and compute its perplexity. Then compute its perplexity on the same question with the words shuffled into a meaningless order. If the perplexity gap is much larger than for fresh, never-seen text of the same length, the model is recognising the eval question — strong evidence it was contaminated even if no n-gram match was found.

Concretely, define the "canary score" $s = \text{ppl}(\text{shuffled}) - \text{ppl}(\text{original})$. For genuinely unseen text, $s$ is small and stable. For contaminated text, $s$ spikes — the model has learned the original ordering. This catches paraphrased leaks that the n-gram filter missed. It is expensive (one full forward pass per eval item) but feasible because the eval suite is small.

Engineering Reality and Gotchas

Contamination detection looks like a clean algorithmic problem. Five production failure modes earn their flags:

1. Short eval items slip past long n-grams. A 12-token MMLU question cannot produce any 13-grams — the sliding window requires the document to be at least $n$ tokens long. The fix is a dual filter: 13-gram for long-form benchmarks, 8-gram for short-form. Skip this and roughly 20% of MMLU and TriviaQA items become undetectable.
2. Tokenisation mismatch between scanner and model. The contamination scanner uses whitespace tokenisation; the model uses BPE. If the scanner ever switches to BPE, the n-gram windows shift, the index is invalidated, and the same training corpus produces different verdicts on different scanner versions. ALWAYS use plain whitespace tokenisation for contamination — it is the stable, model-independent surface form.
3. Eval-set leakage into SFT and RLHF data. The pretraining contamination scan is necessary but not sufficient. SFT instruction datasets and RLHF preference datasets are frequently constructed by paraphrasing public benchmarks — a tutor-model rephrasing GSM8K, for instance. Run the same contamination scanner against every SFT and RLHF batch before shipping. (See section 13.2 for the SFT-specific version.)
4. Benchmark drift. Public benchmarks add and remove items over time. The contamination filter must be REBUILT every time the eval suite changes — a stale filter passes new benchmark items through without checking them. Tie filter rebuild to a hash of the eval suite version; refuse to start training if the hash does not match the live eval suite.
5. Synthetic data contamination by inheritance. Section 8.5 discussed synthetic data — training problems generated by another LLM. If the source LLM was itself contaminated, the synthetic data inherits that contamination, often in a form that surface n-gram filters cannot detect (the leaked content has been paraphrased by a strong model). Always run model-based detection on synthetic data, not just n-gram filtering.

The one sentence to carry forward: a model's benchmark numbers are only as honest as its contamination filter, and an honest contamination filter is cheaper than half a percent of one training run — which is why every serious lab in 2025 ships a contamination appendix alongside its release report.