The Origin of Attention

Biological Attention

Before the attention mechanism was ever written in code, it was written in neurons. For hundreds of millions of years, biological brains have faced an engineering problem that no amount of raw processing power can solve: the world generates far more information than any brain can process. The solution that evolution converged on — the solution that every animal with a cortex relies on — is attention.

Understanding biological attention is not merely a historical curiosity. It is the conceptual foundation of this entire book. Every mathematical construct you will encounter in the chapters ahead — queries, keys, values, softmax distributions, multi-head processing — has a direct analog in the brain. The transformer was not invented from first principles. It was reverse-engineered from biology, whether its creators knew it or not.

The Brain's Information Crisis

Right now, as you read this sentence, your eyes are transmitting approximately 10 million bits of visual information per second to your brain. Your skin is sending another million bits about temperature, pressure, and texture. Your ears contribute 100,000 bits, your nose and tongue add thousands more. In total, your sensory organs flood your brain with roughly 11 million bits of information every second.

But here is the astonishing fact: you are consciously processing approximately 50 bits per second. Recent research from Caltech by Zheng and Meister (2024), published in Neuron, refined this estimate further, finding that humans think at approximately 10 bits per second — roughly 100 million times slower than the sensory input rate. As Markus Meister put it: “Every moment, we are extracting just 10 bits from the trillion that our senses are taking in.”

This is not a deficiency. It is a design principle. The brain contains 85 billion neurons, with one-third dedicated to higher cognition in the cortex. Individual neurons can transmit more than 10 bits per second. The hardware is not the bottleneck — the bottleneck is in the serial, sequential nature of conscious thought. The brain is like a vast ocean of parallel computation funneled through a single narrow river of conscious awareness.

And that funnel — the mechanism that decides what gets through and what does not — is attention. Attention is fundamentally about resource allocation under limited capacity. It is the brain's answer to the question: given that I cannot process everything, what should I process right now?

The Core Insight: Attention is not about adding more processing power. It is about choosing wisely what to process with the power you have. This is true in brains, and it is true in transformers. A 175-billion-parameter GPT model still computes attention weights to decide which tokens matter most for each prediction.

The Cocktail Party Effect

In 1953, British cognitive scientist Colin Cherry published a landmark paper that gave this principle its most vivid demonstration. He called it the cocktail party effect: the remarkable ability to focus on a single speaker's voice in a room full of conversations.

Cherry designed the dichotic listening task: participants wore headphones receiving a different message in each ear and were asked to “shadow” (repeat aloud) the message in one ear while ignoring the other. The results were revelatory:

• Participants could successfully track one message while ignoring the other.
• They noticed physical changes in the unattended channel — if the speaker changed from male to female, they detected it.
• They could not detect semantic changes — if the language switched from English to German, they did not notice.
• But highly salient stimuli, like hearing their own name, could break through the filter — a finding later confirmed by Moray (1959).

This is precisely what a transformer's attention mask does: it suppresses certain positions entirely (like the unattended ear), while allowing high-relevance signals (like your name) to break through based on their compatibility score with the current query.

The Spotlight Model

In 1980, Michael Posner proposed the most influential metaphor in the history of attention research: attention as a spotlight. In his cueing paradigm, participants fixated on a central cross while a cue arrow flashed briefly, pointing left or right. A target then appeared in one of two boxes.

On valid trials (cue matched target location, 80% of the time), reaction times were faster. On invalid trials (cue mismatched, 20%), reaction times were slower. This difference proved that covert attention — attention without eye movement — could be directed in space like an invisible beam.

Posner identified three operations of the attention spotlight:

1. Disengage from the current focus (parietal cortex)
2. Move the spotlight to a new location (superior colliculus)
3. Engage at the new location (pulvinar nucleus of the thalamus)

This maps directly onto the transformer: at each layer, the attention mechanism disengages from the previous context, moves to compute new compatibility scores across all positions, and engages by concentrating weight on the most relevant tokens. The softmax function is the mathematical spotlight — it takes raw scores and creates a focused distribution where the brightest beam falls on the highest-scoring position.

How the Brain Processes Attention

The spotlight metaphor is useful, but the reality is richer. Attention is not a single operation — it is a distributed circuit involving multiple brain regions working in concert. Understanding this circuit reveals why the transformer architecture has the specific structure it does.

The Thalamus: The Gatekeeper

Almost all sensory information (with the notable exception of smell) must pass through the thalamus before reaching the cortex. For decades, neuroscientists described the thalamus as a “relay station.” But research led by Michael Halassa at MIT revealed something far more interesting: the thalamus is a gatekeeper, and its critical structure is the Thalamic Reticular Nucleus (TRN) — a thin shell of inhibitory neurons wrapping around the thalamus.

Halassa's 2019 work, published in Nature, overturned the spotlight metaphor in a crucial way: the brain does not brighten the light on what matters. It lowers the lights on everything else. When mice needed to prioritize auditory information, the prefrontal cortex instructed the visual TRN to increase its inhibitory activity, suppressing the visual thalamus and stripping away irrelevant visual data.

The complete pathway Halassa's team identified:

Prefrontal Cortex $\rightarrow$ Basal Ganglia $\rightarrow$ Thalamic Reticular Nucleus $\rightarrow$ Thalamus

This is precisely what attention masking does in a transformer. When a causal (autoregressive) model sets future positions to $-\infty$ before the softmax, it is doing what the TRN does: suppressing information that should not influence the current computation. The masking mechanism is the transformer's thalamic gate.

The Attention Circuit

The following interactive diagram shows the brain's attention circuit. Hover over each region to see its role and its corresponding transformer component.

Notice the key insight: attention in the brain is not a single structure but an emergent property of coordinated activity across distant regions. The prefrontal cortex does not “do” attention alone — it orchestrates a network. Similarly, transformer attention is not just the softmax function. It is the coordinated interaction of queries, keys, values, masking, scaling, and projection — a system of parts that produces focused processing.

Top-Down vs. Bottom-Up Attention

In 2002, Maurizio Corbetta and Gordon Shulman published a landmark review in Nature Reviews Neuroscience that identified two partially segregated neural networks for attention:

The ventral network is largely lateralized to the right hemisphere and functions as an interrupt system — it can hijack the dorsal network when something unexpected demands attention (a loud noise, a flash of movement, your name spoken across a room).

This dual-network architecture is echoed in multi-head attention. Different attention heads in a transformer learn to specialize: some heads track syntactic dependencies (goal-directed, like the dorsal network), while others respond to unusual or salient tokens (stimulus-driven, like the ventral network). The multi-head design allows the model to balance focused search and surprise detectionsimultaneously.

Neurons That Listen to Context

Attention does not merely select which information reaches consciousness — it physically changes how neurons respond. Research by Treue and Martinez-Trujillo (1999) and Reynolds and Chelazzi (2004) demonstrated that:

• Attention selectively increases the firing rate of neurons encoding task-relevant features and suppresses neurons encoding irrelevant features.
• A top-down gain factor modulates connection strength — effectively amplifying relevant signals, like turning up the volume on a specific frequency.
• Attention can move and resize receptive fields of individual neurons, shifting their preferred features to match the current task.
• Different attention states are signaled by different mixtures of neurons, indicating selective modulation rather than a uniform global gain.

This is exactly what the attention weights in a transformer do. When token “sat” attends strongly to “cat” (weight = 0.45) and weakly to “the” (weight = 0.05), the resulting output vector is dominated by the representation of “cat” — the “neurons” encoding the verb's subject have been amplified, while the article's representation has been suppressed. The gain factor is the attention weight. The receptive field is the attention pattern.

Memory, Context, and the Seed of Q-K-V

The brain does not process each word, sound, or image in isolation. At every moment, it is running a parallel operation: querying its memory, asking what is relevant right now, and retrieving only what it needs. This interaction between attention and memory is the conceptual seed of the most important abstraction in modern deep learning: the Query-Key-Value mechanism.

Working memory — the cognitive system for temporarily maintaining and manipulating information — was formalized by Alan Baddeley and Graham Hitch in 1974. Their model identified a central executive that controls information flow, allocates limited resources, and is implemented primarily in the prefrontal cortex. George Miller's famous 1956 paper established that short-term memory holds approximately $7 \pm 2$ chunks; more recent work by Cowan (2001) revised this to $4 \pm 1$ chunks.

The interaction between attention and working memory is bidirectional: attention selects what enters working memory, and working memory contents influence what attention prioritizes. When concurrent tasks compete for the same prefrontal neural populations, the ability to represent task-relevant information decreases in proportion to demand — demonstrating that attention and working memory share common neural resources.

The “Bank” Problem

Consider what happens when you read the word “bank.” This single word has at least three distinct meanings: a financial institution, the side of a river, or the act of tilting an aircraft. Your brain resolves this ambiguity in roughly 200 milliseconds, and you typically do not even notice the ambiguity existed. How?

The answer is a process that mirrors the transformer's attention mechanism so closely that it feels like the math was written to describe the brain:

1. The brain generates a query from the surrounding context. If the preceding words are “deposited,” “savings,” and “at,” the query encodes a financial frame.
2. This query is matched against keys — the stored semantic associations for “bank” (financial: money, vault, teller; geographical: river, shore, slope; action: tilt, aircraft, turn).
3. The context-matched key (financial) retrieves the corresponding value— the complete meaning entry for “bank” as a financial institution, along with its associations and connotations.
4. This retrieval happens via theta synchronization (4–8 Hz oscillations) between the hippocampus and prefrontal cortex, with stronger coupling predicting better behavioral outcomes.

Try it yourself in the interactive demo below. Change the context and watch how the “attention” shifts to a different meaning:

This is not a loose analogy. The computational structure is identical:

Remember this table. It is the Rosetta Stone of this book. Every mechanism in every subsequent chapter modifies some part of this pipeline — the scoring function, the masking, the softmax, or how Q, K, V are constructed — but the fundamental logic is always: query the memory, match by relevance, retrieve the content.

From Brains to Algorithms

The story of how biological attention became computational attention is a story of frustration. For decades, researchers tried to teach machines to process language — and for decades, they kept running into the same wall that the brain solved millions of years ago.

Early Approaches

The earliest NLP systems were rule-based: linguists hand-coded grammatical rules and hoped the language would cooperate. It did not. Natural language is ambiguous, irregular, and context-dependent in ways that no finite rule set can capture.

N-gram models offered a statistical alternative. Inspired by John Rupert Firth's 1957 observation that “you shall know a word by the company it keeps,” n-grams predicted the next word based on the preceding $n-1$ words. Bigrams captured word pairs; trigrams captured three-word sequences. But n-grams suffered from a fundamental limitation: they could not capture dependencies beyond their window size. The word “bank” in position 20 of a sentence could not be informed by the word “river” in position 3.

The breakthrough came with distributed representations. In 2003, Yoshua Bengio published “A Neural Probabilistic Language Model,” the first paper to propose learning dense, continuous word vectors within a neural network. Ten years later, Tomas Mikolov's Word2Vec (2013) made word embeddings practical and widely accessible, showing that the vector space captured semantic relationships:$\text{king} - \text{man} + \text{woman} \approx \text{queen}$.

But embeddings alone are static — the word “bank” gets the same vector regardless of context. The brain does not work this way. What was needed was a way to make representations context-dependent.

Recurrent Neural Networks

In 1990, Jeffrey Elman published “Finding Structure in Time” in Cognitive Science, introducing the Simple Recurrent Network. For the first time, a neural network could learn linguistic structure — word categories, sentence boundaries — purely from sequential data. The key innovation was a hidden state that persisted across time steps, creating a form of memory.

But vanilla RNNs suffered from the vanishing gradient problem, formally analyzed by Sepp Hochreiter in 1991 and published with Jürgen Schmidhuber in 1997. During backpropagation through time, gradients are computed as products of matrices at each step. When these matrices consistently have values less than 1, gradients shrink exponentially:

The result: early tokens in a sequence received vanishingly small gradient updates. Their information was effectively invisible during training.

Hochreiter and Schmidhuber's LSTM (Long Short-Term Memory, 1997) addressed this with gated memory cells: an input gate (what to write), a forget gate (what to erase), and an output gate (what to read). LSTMs could learn dependencies spanning 1,000+ timesteps. Kyunghyun Cho's GRU (2014) offered a simpler alternative with just two gates.

Yet even LSTMs could not solve the fundamental architectural problem that lay ahead.

The Bottleneck

In 2014, Ilya Sutskever, Oriol Vinyals, and Quoc V. Le published “Sequence to Sequence Learning with Neural Networks” — the paper that created the modern encoder-decoder paradigm. The idea was elegant: use one LSTM (the encoder) to read the entire input sentence and compress it into a single fixed-length vector, then use another LSTM (the decoder) to generate the output from that vector.

The problem was the vector. Regardless of whether the input sentence had 5 words or 50, the entire meaning had to be compressed into a vector of the same fixed dimensionality — typically 256 to 1024 dimensions. Imagine being asked to listen to an entire lecture, summarize it in a single sentence, and then someone must reconstruct the full lecture from your summary alone. The longer the lecture, the more is lost.

Cho et al. (2014) demonstrated this failure empirically: the encoder-decoder model performed well on short sentences, but quality degraded sharply for sentences longer than 20–30 words. Early tokens were systematically lost because the fixed-capacity vector had a recency bias — later tokens overwrote earlier ones.

Sutskever's team even discovered a clever hack: reversing the input sentence order so that the first output tokens were temporally close to the corresponding input tokens. This improved BLEU scores significantly — a revealing patch that proved the bottleneck was real.

The question that changed everything: What if the decoder didn't have to work from a single compressed vector? What if, at each decoding step, it could look back at all encoder states and decide which ones matter most — the way a human reader re-reads specific words to resolve ambiguity?

The Birth of Attention in Deep Learning

Bahdanau Attention (2014)

The answer came from a 23-year-old intern named Dzmitry Bahdanau, working in Yoshua Bengio's Montreal lab in the summer of 2014. His inspiration was remarkably simple: in middle school translation exercises, your gaze shifts back and forth between the source and target text as you translate. You do not read the entire source once, memorize it, and then write the translation from memory. You look back.

Bahdanau designed a mechanism where the decoder could, at each step, compute a relevance score between its current state and every encoder hidden state. He initially experimented with hard-coded diagonal attention and synchronized cursors, but realized the decoder could learn where to focus through a soft search mechanism using softmax and weighted averaging.

The mechanism worked on the very first try. Bahdanau completed the entire system — from concept to working code to published paper — in approximately five weeks of his remaining internship. During the final revisions, Yoshua Bengio suggested the name “attention.”

The following code shows the core of Bahdanau attention — additive alignment scoring between the decoder state (the query) and all encoder states (the keys and values):

1import numpy as np
2
3# Encoder hidden states (bidirectional RNN output)
4# Shape: (source_length, hidden_dim)
5encoder_states = np.array([
6    [0.1, 0.9, 0.2],   # "The"
7    [0.8, 0.1, 0.3],   # "European"
8    [0.7, 0.2, 0.8],   # "Commission"
9    [0.3, 0.6, 0.1],   # "said"
10])
11
12# Current decoder hidden state
13decoder_state = np.array([0.5, 0.4, 0.7])
14
15# Learned weight matrices (simplified)
16W_a = np.random.randn(3, 3) * 0.3
17U_a = np.random.randn(3, 3) * 0.3
18v_a = np.random.randn(3) * 0.3
19
20# Step 1: Compute alignment scores
21scores = []
22for h_i in encoder_states:
23    # Additive attention: v^T * tanh(W*s + U*h)
24    energy = np.tanh(W_a @ decoder_state + U_a @ h_i)
25    score = v_a @ energy
26    scores.append(score)
27
28scores = np.array(scores)
29
30# Step 2: Softmax to get attention weights
31exp_scores = np.exp(scores - np.max(scores))
32weights = exp_scores / exp_scores.sum()
33
34# Step 3: Weighted sum of encoder states
35context = np.sum(
36    weights[:, np.newaxis] * encoder_states,
37    axis=0
38)
39
40print("Attention weights:", weights.round(3))
41print("Context vector:", context.round(3))

The results were immediate and dramatic. Bahdanau's “RNNSearch” model achieved higher BLEU scores than the fixed-bottleneck approach, especially on long sentences. More remarkably, the attention weights produced interpretable soft alignments between source and target words, showing that the model had learned linguistically meaningful correspondences without being told about word alignment.

As Bengio later reflected: “My own insight really became strong in the context of the machine translation task. Prior to our introduction of attention, we were using a recurrent network that read the whole input source language sequence and then generated the translated target language sequence. However, this is not at all how humans translate. Humans pay very particular attention to just one word or a few input words at a time.”

From Alignment to Scaled Dot-Product

Bahdanau's additive attention used a learned feedforward network to compute scores:$\text{score}(s_t, h_i) = v_a^\top \tanh(W_a[s_t; h_i])$. This was expressive but slow, requiring matrix multiplications and a nonlinear activation for every query-key pair.

In 2015, Thang Luong, Hieu Pham, and Christopher Manning at Stanford proposed a simpler alternative: multiplicative attention. Instead of a feedforward network, they used a simple dot product: $\text{score}(s_t, h_i) = s_t^\top h_i$. No learned parameters, no nonlinearity — just a dot product. This achieved a 5.0 BLEU point improvement over non-attentional systems and set the stage for the final breakthrough.

The evolution is clean:

Each step simplified the mechanism while preserving its power. The scaling factor$\frac{1}{\sqrt{d_k}}$ in the final form was critical: without it, for large $d_k$, dot products grow large in magnitude, pushing softmax into regions where gradients become extremely small. This insight came from Noam Shazeer.

The Transformer (2017)

On June 12, 2017, eight Google researchers — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin — published “Attention Is All You Need.” The title, a nod to the Beatles' “All You Need Is Love,” carried a revolutionary claim: attention mechanisms alone, without any recurrence or convolution, are sufficient for state-of-the-art sequence transduction.

The key innovations:

1. Self-Attention: Each position in a sequence attends to every other position, enabling direct modeling of all pairwise relationships regardless of distance. No more vanishing gradients over long distances.
2. Multi-Head Attention: Instead of single attention, the model runs $H$ parallel attention functions, each with different learned projections. This allows simultaneously attending to syntax, semantics, and position — like the brain's parallel dorsal and ventral streams.
3. Scaled Dot-Product: The $QK^\top / \sqrt{d_k}$ formula, with the scaling factor ensuring stable training.
4. Positional Encoding: Since attention is permutation-invariant (unlike RNNs, which process tokens in order), sinusoidal positional encodings inject sequence order information.
5. Full Parallelization: No sequential dependencies means massive speedups on GPU hardware. The base model trained in just 12 hours on 8 GPUs.

The following code shows the scaled dot-product attention — the formula at the heart of every transformer:

1import numpy as np
2
3def scaled_dot_product_attention(Q, K, V):
4    """The formula that changed everything.
5
6    Attention(Q, K, V) = softmax(Q @ K.T / sqrt(d_k)) @ V
7    """
8    d_k = K.shape[-1]
9
10    # Step 1: Dot product of queries and keys
11    scores = Q @ K.T
12
13    # Step 2: Scale by sqrt(d_k)
14    scores = scores / np.sqrt(d_k)
15
16    # Step 3: Softmax to get attention weights
17    exp_scores = np.exp(scores - scores.max(axis=-1, keepdims=True))
18    weights = exp_scores / exp_scores.sum(axis=-1, keepdims=True)
19
20    # Step 4: Weighted sum of values
21    output = weights @ V
22
23    return output, weights
24
25# Example: "The cat sat on the mat"
26# Each row is a token's learned representation
27Q = np.array([[1, 0, 1, 0],    # "The" asks: what's relevant to me?
28              [0, 2, 0, 1]])    # "cat" asks: what's relevant to me?
29
30K = np.array([[0, 1, 0, 1],    # "The" advertises its features
31              [1, 0, 1, 0]])    # "cat" advertises its features
32
33V = np.array([[1, 0, 0, 0],    # "The" offers this content
34              [0, 1, 0, 0]])    # "cat" offers this content
35
36output, weights = scaled_dot_product_attention(Q, K, V)
37print("Attention weights:\n", weights.round(3))
38print("Output:\n", output.round(3))

The results were decisive. On WMT 2014 English-to-German translation, the transformer achieved 28.4 BLEU, improving over existing best systems (including ensembles) by over 2 points. On English-to-French, it set a new single-model record of 41.8 BLEU.

Jakob Uszkoreit proposed the core idea of replacing RNNs with self-attention and chose the name “Transformer” simply because he liked the sound of it. Even his father, renowned computational linguist Hans Uszkoreit, was initially skeptical. An early design document was titled “Transformers: Iterative Self-Attention and Processing for Various Tasks” and featured illustrations from the Transformers franchise.

The Unified Mental Model

We have now traced a single thread from the cocktail party in 1953 to the transformer in 2017. The thread is this: attention is the brain's solution to resource allocation under limited capacity, and the transformer is the first computational architecture that captures this solution completely.

The following diagram maps every biological component we have discussed onto its transformer counterpart. This is the mental model you should carry into every subsequent chapter:

The side-by-side comparison below makes the parallel visceral. On the left, the brain's spotlight mechanism — a cone of enhanced processing directed at the most relevant stimulus. On the right, the softmax attention distribution — a probability distribution that concentrates weight on the most relevant tokens. Same problem, same solution, different substrates.

This is not merely a pedagogical convenience. In January 2024, Kozachkov et al. published “Short-term Hebbian learning can implement transformer-like attention” in PLOS Computational Biology, demonstrating that biological neurons can implement attention-like computations through short-term Hebbian synaptic potentiation. The study found that:

• Somatic spike trains (queries) backpropagate into apical dendrites, where they are compared with axonal activity (keys) via NMDA receptor calcium influx.
• Spines receiving matched inputs undergo 8-fold increases in synaptic strength — the biological equivalent of a high attention weight.
• The integrated calcium signal approximates a bilinear overlap between spike trains that resembles the dot-product similarity in transformer attention.
• The kernel model explained 81% of the variance in attention behavior.

The transformer is not an invention. It is a rediscovery. Engineers reverse-engineered something the brain has been doing for millions of years.

The Interactive Spotlight

To make this tangible, the demo below lets you type any sentence and observe how a simulated attention mechanism allocates weights. Click on any word to make it the “query,” and watch which other words receive the most attention. Adjust the temperature to see how it affects the sharpness of focus.

Summary: The Translation Table

Carry this table into every subsequent chapter. It maps each biological concept introduced in this chapter to its mathematical counterpart in the transformer. When you encounter a formula that seems abstract, look it up here and remember: it formalizes something your brain already knows how to do.

The takeaway: The transformer did not spring from pure mathematics. It is the formalization of a biological principle that evolution discovered hundreds of millions of years ago: the brain that best allocates its limited processing resources to the most relevant information is the brain that survives.

In the next chapter, we fix a single example sentence and its Q, K, V matrices, then begin the mathematical journey. You will see the formula $\text{softmax}(QK^\top / \sqrt{d_k})V$ applied step by step to real numbers. But now, when you see that formula, you will recognize it for what it is: a compact description of something your brain does every time you read a sentence, hear your name in a crowd, or reach for the right meaning of an ambiguous word.

References

Neuroscience of Attention

1. Cherry, E. C. (1953). “Some Experiments on the Recognition of Speech, with One and with Two Ears.” Journal of the Acoustical Society of America, 25(5), 975–979.
2. Broadbent, D. E. (1958). Perception and Communication. London: Pergamon Press.
3. Treisman, A. M. (1964). “Selective Attention in Man.” British Medical Bulletin, 20, 12–16.
4. Posner, M. I. (1980). “Orienting of Attention.” Quarterly Journal of Experimental Psychology, 32(1), 3–25.
5. Norretranders, T. (1998). The User Illusion: Cutting Consciousness Down to Size. Penguin Press.
6. Corbetta, M., & Shulman, G. L. (2002). “Control of Goal-Directed and Stimulus-Driven Attention in the Brain.” Nature Reviews Neuroscience, 3, 201–215.
7. Treue, S., & Martinez-Trujillo, J. C. (1999). “Feature-Based Attention Influences Motion Processing Gain in Macaque Visual Cortex.” Nature, 399, 575–579.
8. Reynolds, J. H., & Chelazzi, L. (2004). “Attentional Modulation of Visual Processing.” Annual Review of Neuroscience, 27, 611–647.
9. Halassa, M. M. et al. (2019). “Thalamocortical control of cognition.” Nature Reviews Neuroscience.
10. Zheng, J., & Meister, M. (2024). “Thinking Slowly: The Paradoxical Slowness of Human Behavior.” Neuron.

Memory and Language Processing

1. Miller, G. A. (1956). “The Magical Number Seven, Plus or Minus Two.” Psychological Review, 63(2), 81–97.
2. Baddeley, A. D., & Hitch, G. (1974). “Working Memory.” In G.H. Bower (Ed.), The Psychology of Learning and Motivation, 8, 47–89.
3. Cowan, N. (2001). “The Magical Number 4 in Short-Term Memory.” Behavioral and Brain Sciences, 24(1), 87–114.
4. Rodd, J. M. et al. (2005). “The Neural Mechanisms of Speech Comprehension: fMRI Studies of Semantic Ambiguity.” Cerebral Cortex, 15(8), 1261–1269.
5. Preston, A. R., & Eichenbaum, H. (2013). “Interplay of Hippocampus and Prefrontal Cortex in Memory.” Current Biology, 23(17), R764–R773.

Sequence Modeling and Neural Networks

1. Elman, J. L. (1990). “Finding Structure in Time.” Cognitive Science, 14(2), 179–211.
2. Hochreiter, S., & Schmidhuber, J. (1997). “Long Short-Term Memory.” Neural Computation, 9(8), 1735–1780.
3. Bengio, Y. et al. (2003). “A Neural Probabilistic Language Model.” Journal of Machine Learning Research, 3, 1137–1155.
4. Mikolov, T. et al. (2013). “Efficient Estimation of Word Representations in Vector Space.” arXiv:1301.3781.
5. Cho, K. et al. (2014). “Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation.” arXiv:1406.1078.
6. Sutskever, I., Vinyals, O., & Le, Q. V. (2014). “Sequence to Sequence Learning with Neural Networks.” Advances in NeurIPS, 27.

Attention Mechanisms

1. Bahdanau, D., Cho, K., & Bengio, Y. (2015). “Neural Machine Translation by Jointly Learning to Align and Translate.” Proceedings of ICLR 2015. arXiv:1409.0473.
2. Luong, T., Pham, H., & Manning, C. D. (2015). “Effective Approaches to Attention-based Neural Machine Translation.” Proceedings of EMNLP 2015. arXiv:1508.04025.
3. Vaswani, A. et al. (2017). “Attention Is All You Need.” Advances in NeurIPS, 30. arXiv:1706.03762.

Biology-Transformer Connection

1. Milner, A. D., & Goodale, M. A. (1992). “Two Visual Systems Re-viewed.” Neuropsychologia.
2. Kozachkov, L. et al. (2024). “Short-term Hebbian learning can implement transformer-like attention.” PLOS Computational Biology, 20(1), e1011843.