Quality Filtering

Section 8.2 took a 15-trillion-token raw web crawl and removed the duplicates. What remained was still mostly junk. The web is unfiltered in a way that does not survive contact with a learning machine: navigation bars, machine-translated nonsense, autogenerated SEO farms, broken Unicode, copy-pasted boilerplate, link spam, and pages whose only content is a cookie banner. Train a model on the deduped crawl as-is and you will burn months of GPU time learning the statistics of garbage. Quality filtering is the engineering practice that decides which documents are worth a model's attention — and at the 14.8T-token scale of DeepSeek-V3, it is the single largest lever on final model quality short of architecture itself.

What this section delivers. The three-stage cascade every modern pretraining stack uses — heuristic rules, perplexity scoring under a reference language model, and a learned binary classifier — derived from the problem each stage solves, computed on a toy corpus by hand, implemented in plain Python and then PyTorch, and finally connected to the engineering reality of running it on a petabyte of raw crawl per week.

The Junk Problem at 15 Trillion Tokens

Common Crawl serves a snapshot of the open web every few weeks. A single snapshot is about 3.5 PB compressed, roughly 90B documents, and somewhere on the order of 50T tokens after a permissive HTML→text extraction. After the dedup pass from section 8.2 the corpus shrinks by about $3\times$ — but the surviving 15T tokens are still, by document-level eyeballing, only 15–25% actually useful for teaching a language model.

The other 75–85% is not random noise. It is structured noise: real strings that real humans (or scrapers) put on the web for reasons that have nothing to do with conveying knowledge. Three families dominate.

The cost of not filtering is not subtle. The original GPT-3 paper notes a roughly $2\times$ compute equivalence between adding curated data and growing the model — every junk token consumed was a model token of capacity wasted. Llama 2 reports that the Wikipedia-classifier filter contributed more to final downstream MMLU score than doubling the deduplication aggressiveness did. DeepSeek-V3 explicitly attributes a large share of its punching-above-its-active-parameter-count to aggressive quality filtering, not architecture.

Intuition: A Cascade of Increasingly Expensive Sieves

The temptation is to write one big “is this doc good?” function. Resist it. Quality filtering is built as a cascade of independent stages because the cost-per-doc rises by orders of magnitude as the signal-per-doc rises. Each stage's job is to make the next stage cheaper.

The cascade is what makes the budget arithmetic close. Run the classifier on every raw doc and you spend $5 \text{ ms} \times 90\text{B} \approx 14$ million CPU-core-hours per snapshot. Run heuristics first to filter out 80% of the crawl, then perplexity to remove another 50%, and the classifier only sees $90\text{B} \cdot 0.2 \cdot 0.5 = 9\text{B}$ docs — a budget that fits on a single CPU cluster overnight. Re-ordering the cascade is the difference between a tractable pipeline and an intractable one.

The Math of a Quality Filter

Each stage produces a real-valued score for every document. Let $d$ be a document and let $s_1(d), s_2(d), s_3(d)$ be the heuristic, perplexity, and classifier scores. The pipeline applies three thresholds $\tau_1, \tau_2, \tau_3$ and keeps the document iff all three pass:

$\text{keep}(d) = \mathbf{1}\!\left[s_1(d) \geq \tau_1\right] \cdot \mathbf{1}\!\left[s_2(d) \leq \tau_2\right] \cdot \mathbf{1}\!\left[s_3(d) \geq \tau_3\right]$

Two of the inequalities point the way you expect (more of this is better) and one (perplexity) points the other way (lower perplexity = more reference-like = better). $\mathbf{1}[\cdot]$ is the indicator function — 1 if the predicate is true, 0 otherwise.

Stage 1: heuristic score

The simplest workable form is the alphanumeric ratio, $s_1(d) = |\{c \in d : c\text{ is letter}\}| / |d|$. Real prose lands at $s_1 \in [0.75, 0.85]$; nav-bar dumps land near $0.4$. A threshold of $\tau_1 = 0.6$ is the standard starting point. Production pipelines stack a dozen such rules (mean word length, line repetition rate, ratio of bullet points, fraction of lines ending in terminal punctuation, etc.). Each is a single O(N) string pass and runs before anything else.

Stage 2: perplexity under a reference LM

Train a small n-gram language model $p_{\text{ref}}$ on a curated reference corpus (Wikipedia + books works well). Score document $d = w_1 w_2 \dots w_N$ with its perplexity, which is the exponentiated average negative log-probability per word:

$s_2(d) = \text{PP}(d) = \exp\!\left(-\frac{1}{N} \sum_{i=1}^{N} \log p_{\text{ref}}(w_i \mid w_{i-1}, \dots)\right)$

Interpretation. Perplexity is the “effective vocabulary size” the reference LM perceives at each step — a low $\text{PP}$ means the LM is consistently unsurprised by the next word, i.e. the document is similar to what the LM was trained on. A high $\text{PP}$ means the LM is constantly surprised — the document either uses words the reference never saw (gibberish, foreign language, code-switched text) or strings them together in ways the reference never saw (machine translation, autogen). The threshold $\tau_2$ is set per-domain because domain-specific text legitimately scores higher under a Wikipedia LM than Wikipedia does.

Stage 3: learned binary classifier

Train a fastText-style classifier with reference text (Wikipedia + a few curated sources) as the positive class and an unfiltered random crawl as the negative class. The model is a single embedding table with mean-pooling and a 2-class linear head — about 10M parameters. Score documents by the predicted probability of the “reference” class:

$s_3(d) = P\!\left(\text{class} = \text{ref} \mid d ; \theta\right) = \sigma\!\left(W \cdot \frac{1}{N} \sum_{i=1}^{N} E[w_i]\right)$

where $E \in \mathbb{R}^{V \times d}$ is the embedding table, $W \in \mathbb{R}^{2 \times d}$ is the classifier head, and $\sigma$ is the softmax. The classifier picks up register and topic signals that unigram perplexity cannot — it distinguishes “textbook-quality explanation” from “competent blog post” from “auto-spun marketing copy.”

Why the indicator-product form is exact, not a heuristic

Treating the three stages as independent thresholds is a deliberate modelling choice. You could imagine a soft-AND like $s_1 + s_2' + s_3 > \tau$ instead. Production pipelines do not, for two reasons. First, the independence assumption maps cleanly to the cascade ordering: each stage's threshold can be tuned in isolation against a held-out labelled set. Second, hard rejection at each stage is what makes the cost arithmetic work — soft-AND would force every stage to run on every document.

Manual Numerical Walkthrough

Five documents. Three filter stages. By hand. The point is to see that the numbers for the cascade are not mysterious — they are mean-ratios, log-sums, and one matrix-vector product.

Visualizing the Three-Stage Pipeline

The visualizer below runs a synthetic 160-document corpus through the same three-stage cascade. Each bubble is one document; the bubble radius is the doc's token count. Drag any of the three threshold sliders and watch the cohort of survivors (green) shrink or expand, and the cohorts dropped at each stage (red / amber / violet) flow into the trash zone. The stats panel reports the same numbers a production pipeline logs after every shard.

Three behaviours to lock in by playing with the sliders. First, drop the heuristic threshold to 0.3 — most boilerplate-looking docs leak through, and the average quality of survivors falls sharply. Second, drop the perplexity cutoff from 110 to 50 (much stricter) — token retention craters and the kept corpus becomes uniformly Wikipedia-like, which is precisely the “mode collapse” failure mode mentioned in the next-to-last section. Third, raise the classifier threshold above 0.7 — the kept pile becomes tiny but its average quality reaches the ceiling. There is no setting that simultaneously maximizes retention and quality. The job is to find the right point on that frontier for the downstream model's objective.

Plain Python: Heuristic + Perplexity Filter

Both stages of the cheap half of the cascade in self-contained Python — no PyTorch, no KenLM. The reference LM is a unigram trained on three sentences, the heuristic is a handful of guard clauses, and the pipeline is a guard-clause function that returns both the keep / drop decision and the reason. Run it on five toy documents to see every branch fire.

1import math, re
2from collections import Counter
3
4# ------------------------------------------------------------------
5# A tiny "reference distribution" built from a clean corpus (Wiki-like).
6# In production this is a 5-gram KenLM trained on Wikipedia or books.
7# Here we hand-build a unigram for clarity.
8# ------------------------------------------------------------------
9REFERENCE = (
10    "the quick brown fox jumps over the lazy dog "
11    "deep learning is the study of neural networks "
12    "transformers replaced recurrent models for language tasks"
13).split()
14ref_counts = Counter(REFERENCE)
15V = sum(ref_counts.values())
16ref_prob = {w: c / V for w, c in ref_counts.items()}
17
18# ------------------------------------------------------------------
19# Stage 1: HEURISTIC FILTER -- cheap, runs first, drops 60-80% of crawl.
20# ------------------------------------------------------------------
21def heuristic_ok(text: str) -> bool:
22    if len(text) < 50:                    # too short to be a real doc
23        return False
24    alpha = sum(c.isalpha() for c in text) / max(1, len(text))
25    if alpha < 0.6:                       # mostly punctuation / nav cruft
26        return False
27    words = text.split()
28    if len(words) < 10 or len(words) > 100000:
29        return False
30    avg_wlen = sum(len(w) for w in words) / len(words)
31    if not (3 <= avg_wlen <= 10):         # English words are ~5 chars on avg
32        return False
33    return True
34
35# ------------------------------------------------------------------
36# Stage 2: PERPLEXITY FILTER -- score doc under the reference LM.
37# Lower perplexity = more "reference-like" = keep.
38# ------------------------------------------------------------------
39def doc_perplexity(text: str, smooth: float = 1e-4) -> float:
40    words = re.findall(r"[a-z]+", text.lower())
41    if not words:
42        return float("inf")
43    logp = 0.0
44    for w in words:
45        p = ref_prob.get(w, smooth)       # OOV gets a tiny floor
46        logp += math.log(p)
47    return math.exp(-logp / len(words))   # geometric mean of 1/p
48
49PERPLEXITY_CUTOFF = 60.0
50
51# ------------------------------------------------------------------
52# The pipeline. Each filter is a guard clause; cheap stages run first
53# so we never spend the expensive perplexity computation on obvious junk.
54# ------------------------------------------------------------------
55def keep(text: str) -> tuple[bool, str]:
56    if not heuristic_ok(text):
57        return False, "heuristic"
58    if doc_perplexity(text) > PERPLEXITY_CUTOFF:
59        return False, "perplexity"
60    return True, "kept"
61
62corpus = [
63    "Deep learning is the study of neural networks trained on data.",
64    "BUY NOW!!! Click here >>> http://spam.tld/win <<<",
65    "the quick brown fox jumps over the lazy dog repeatedly forever",
66    "asdf qwer zxcv 1234 ////  ::::  ____  random gibberish line",
67    "Transformers replaced recurrent models for language tasks in 2017.",
68]
69
70for doc in corpus:
71    kept, reason = keep(doc)
72    ppl = doc_perplexity(doc)
73    print(f"{'KEEP' if kept else 'DROP':4s} | reason={reason:10s} "
74          f"| ppl={ppl:8.2f} | {doc[:55]}")

Two implementation choices in this code generalise to every production pipeline. First, every stage returns both a decision and a reason. The reasons accumulate into a histogram per shard and per source domain, and that histogram is what the data team stares at every morning. Second, the perplexity computation works in log space and uses a smoothing floor — without those two details, real documents would either underflow to zero probability or return NaN on the first OOV token.

Sanity check. If you replace the unigram with a 5-gram KenLM trained on 5B Wikipedia words, the same five-doc corpus produces very different perplexity numbers but the same relative ordering. The cascade's correctness depends on the ordering, not on the absolute scores — which is why every team re-tunes thresholds the moment they swap reference LMs.

PyTorch: Batched fastText-Style Classifier Scoring

The expensive third stage. A fastText-shaped classifier — embedding table + mean-pool + linear head — scored in batches on the GPU. The function returns one score per document; the threshold is applied by the caller. Variable-length docs are padded to the longest in each batch and a boolean mask keeps pad tokens from polluting the mean pool.

1import torch
2import torch.nn.functional as F
3
4@torch.no_grad()
5def quality_score_batch(
6    classifier: torch.nn.Module,   # tiny linear: (V,) -> (2,) reference vs crawl
7    token_id_batches: list[torch.Tensor],  # list of (L_i,) int64 tensors
8    device: str = "cuda",
9    batch_size: int = 2048,
10) -> torch.Tensor:
11    """Returns one quality score in [0,1] per document.
12
13    The 'classifier' is fastText-style: a single embedding table summed
14    across the document, fed through one linear layer, softmaxed.
15    Trained to discriminate 'reference' (Wikipedia / curated) from
16    'random crawl'. P(reference) becomes the quality score.
17    """
18    scores = []
19    for start in range(0, len(token_id_batches), batch_size):
20        batch = token_id_batches[start:start + batch_size]
21
22        # Pad to the longest doc in this batch. Padding token = 0.
23        max_len = max(t.size(0) for t in batch)
24        padded = torch.zeros(len(batch), max_len, dtype=torch.long, device=device)
25        mask   = torch.zeros(len(batch), max_len, dtype=torch.bool, device=device)
26        for i, t in enumerate(batch):
27            padded[i, :t.size(0)] = t.to(device)
28            mask[i, :t.size(0)]   = True
29
30        # Mean-pooled bag-of-tokens embedding (the fastText trick).
31        emb = classifier.embed(padded)                  # (B, L, d)
32        emb = emb * mask.unsqueeze(-1)                  # zero out pads
33        doc_emb = emb.sum(dim=1) / mask.sum(dim=1, keepdim=True).clamp(min=1)
34
35        logits = classifier.head(doc_emb)               # (B, 2)
36        p_ref  = F.softmax(logits, dim=-1)[:, 1]        # P(reference)
37        scores.append(p_ref.cpu())
38
39    return torch.cat(scores, dim=0)
40
41# ----- usage at scale: ~200M docs across one node ----------------
42# token_id_batches comes from the streaming tokenizer (Chapter 3).
43# Threshold is tuned on a held-out 50k-doc validation slice where
44# humans labeled "kept this for pretraining? yes/no".
45scores = quality_score_batch(classifier, token_id_batches)
46keep_mask = scores >= 0.55      # 0.55 = the third slider in the viz
47kept = [d for d, k in zip(documents, keep_mask.tolist()) if k]
48print(f"kept {len(kept):,} of {len(documents):,}  "
49      f"({100*len(kept)/len(documents):.1f}%)")

Three subtleties about how this function plugs into the rest of the pretraining stack are worth marking explicitly.

1. Token IDs in, scores out — text is never reprocessed. The streaming tokenizer (chapter 3) is the only component that touches raw text. Every downstream filter and the model itself consumes int64 IDs. This decoupling means swapping classifiers re-runs in hours, not days.
2. The classifier is tiny on purpose. ~10M parameters, one matmul per doc. A bigger classifier (a small transformer) would score better per doc but would dominate the pipeline cost — for minimal gain because by stage 3 the surviving documents are already mostly OK. The right place to spend capacity is the model being trained, not the gatekeeper at the door.
3. The 0.55 threshold is empirical, not principled. It is the value that maximised downstream MMLU/HumanEval/GSM8K on a scaled-down 1B-parameter pilot. Every model size and every downstream eval suite has its own optimum. Production teams re-tune this threshold per snapshot, sometimes per domain.

At Massive Scale: 15T → 14.8T Tokens

Numbers from public reports. None of the major frontier models publish the exact filter settings, but the order-of-magnitude shape of the funnel is consistent across DeepSeek-V3, Llama 3, RedPajama-V2, and Falcon.

Two things to read from the table. First, every stage roughly thirds the corpus. Compounded over the three quality stages that is a $0.7^3 \approx 0.34$ retention from deduped input to final corpus — almost two-thirds of the deduped tokens are rejected for quality reasons. Second, the compute is dominated by the dedup pass and the perplexity stage, not the classifier. The cascade ordering pays for itself by keeping the classifier's bill under one percent of total pipeline cost.

The 14.8T number comes from many snapshots, not one

A single Common Crawl snapshot yields ~5T tokens after quality filtering. DeepSeek-V3 reports 14.8T training tokens, which comes from accumulating across multiple snapshots plus curated sources (the next-section topic of data mixing). The filtered output of one snapshot is not enough; the filtered output of many snapshots is what feeds a frontier-scale run.

Why filter quality matters more for MoE than for dense

Mixture-of-experts models (chapter 5) have many experts, each specialising on a slice of the input distribution. A junk-heavy training set creates a junk-handling expert that the router learns to send certain inputs to — and that expert's capacity is now permanently wasted. Dense models have no such routing; junk is averaged into every weight. As a result MoE models are more sensitive to data quality at fixed token count: the DeepSeek paper explicitly notes that tightening the classifier threshold by 5% gave them a larger downstream-eval bump than a comparable dense baseline saw.

Filtering interacts with the tokenizer

Your reference LM and your classifier both score documents using a tokenization that is decoupled from the main-model tokenizer (chapter 3). When you change the main-model tokenizer mid-project (rare but it happens) the filter scores do not automatically change with it — you are filtering on the old distribution and training on the new one. Teams have caught this exact bug 6 months into a run when downstream scores plateau unexpectedly; the fix is to re-score the entire corpus with a classifier rebuilt under the new tokenization.

Engineering Reality and Failure Modes

The cascade looks tidy in writing and is anything but in practice. Five failure modes recur across every team that has shipped a pretraining pipeline.

1. Mode collapse toward the reference distribution. Over-strict perplexity cutoffs (or a classifier with too narrow a reference) produce a corpus that looks uniformly like Wikipedia. The model trained on it sounds like an encyclopaedia and fails on conversational, code, and creative tasks. Fix: per-domain cutoffs, and a small explicit budget of “diverse” data that bypasses stage 3.
2. Classifier label drift. The classifier is trained on a fixed pair of (reference, crawl) sets. The crawl distribution drifts every quarter (new platforms, new spam formats). After a year, the “crawl” class no longer represents the current crawl, and the classifier's decisions become unreliable in ways that are invisible to its accuracy on the original validation set. Fix: retrain the classifier per snapshot, monitor score distributions for distribution shift.
3. Language-specific failures. A perplexity LM trained on English Wikipedia gives nonsensically high perplexity to perfectly good Mandarin or Hindi prose. If language ID is buggy upstream, you silently delete entire languages. Fix: a separate reference LM per language, gated by a language-ID classifier upstream — and per-language retention dashboards that scream when a language drops off a cliff.
4. Eval contamination via the reference set. If your reference corpus contains MMLU questions (because Wikipedia does), the classifier learns to prefer documents that look like MMLU questions, and your training set ends up enriched for MMLU-leakage patterns. Section 8.7 covers contamination detection; the relevant rule here is to scrub the reference corpus of any eval data before using it as the classifier's positive class.
5. The “long tail of legitimately weird” problem. Source code, math papers with heavy LaTeX, chemistry formulas, and tabular data all score badly under a prose-trained reference LM but are exactly the domains the downstream model needs to learn. The fix is the next section's topic: domain-aware data mixing, where each domain has its own filter and its own weighting in the final training mixture. Quality filtering is per-domain; data mixing is how the per-domain corpora are combined.

Past all these mechanics, the deeper lesson of quality filtering is that the pretraining model never sees raw reality. It sees the world through the filter cascade your team built — and every choice you make in this section is permanently encoded in what the model thinks “good text” means. The filter is the model's upbringing. Treat it accordingly.

What the next section adds. Section 8.4 Data Mixing and Domain Weighting takes the filtered corpora produced here (per-domain: web, code, math, books, news, multilingual) and answers the question of how much of each to feed the model in what order. Quality filtering decides what each domain looks like; data mixing decides how loudly each domain speaks.