From RNNs to GPT: How “Attention Is All You Need” Sparked a Revolution in NLP

The field of natural language processing (NLP) has undergone a dramatic transformation over the past decade, evolving from the sluggish Recurrent Neural Networks (RNNs) to the powerful GPT (Generative Pre-trained Transformer) models that power modern AI applications. At the heart of this revolution lies a single paper: “Attention Is All You Need” (Vaswani et al., 2017), which introduced the Transformer architecture. In this technical blog, we’ll trace the journey from RNNs to the Transformer and finally to GPT, focusing on how the Transformer’s innovations—particularly self-attention and vector encoding/decoding—paved the way for GPT’s creation. We’ll use a simple example and analogies to make this accessible to non-technical readers, while providing enough detail for those familiar with NLP.

The Starting Point: RNNs and Their Limitations

Our journey begins in the early 2010s, when RNNs, including their advanced variants like Long Short-Term Memory (LSTM) networks, were the go-to models for NLP tasks like machine translation and text generation. RNNs process sequences (e.g., sentences) one word at a time, maintaining a “memory” of prior words through hidden states. Picture an RNN as a librarian reading a book aloud, word by word, trying to keep the entire story in mind.

However, RNNs faced critical challenges:

1. Sequential Processing: RNNs process words one after another, preventing parallel computation. This is like the librarian refusing to skim ahead, making training slow and inefficient for long texts.
2. Vanishing Gradients: During training, RNNs struggle to learn relationships between words far apart in a sentence because gradients (signals used to update the model) fade over long sequences. It’s as if the librarian forgets the story’s beginning by the time they reach the end.
3. Limited Context: Even LSTMs, designed to retain longer memories, struggle with very long sequences, losing critical context for complex tasks.

These limitations made RNNs ill-suited for scaling to large datasets or handling tasks requiring deep contextual understanding, setting the stage for a new approach.

The Turning Point: “Attention Is All You Need” and the Transformer

In 2017, Google Research published the seminal paper “Attention Is All You Need,” introducing the Transformer architecture. The Transformer abandoned RNNs entirely, relying instead on a mechanism called self-attention to process sequences. Unlike RNNs, it processes all words in a sentence simultaneously, enabling faster training and better handling of long-range dependencies.

Self-Attention: The Core Innovation

Self-attention allows the model to weigh the importance of each word in a sentence when processing any given word, regardless of its position. The paper introduced scaled dot-product attention, which computes these weights efficiently using matrix operations. Formally, self-attention calculates a weighted sum of input representations, where weights are determined by the relevance of each word to others.

Analogy: Imagine you’re at a networking event, trying to follow a conversation. Instead of listening to one person at a time (like an RNN), you scan the room, focusing on who’s saying what and how their words relate. You might prioritize the keynote speaker over background chatter, dynamically adjusting your attention. Self-attention works similarly, assigning weights to words based on their relevance to create context-aware representations.

Vector Encoding and Decoding in the Transformer

The Transformer processes text through two main components: the encoder (for input processing) and the decoder (for output generation). Both rely on vector encoding and decoding:

• Vector Encoding: Words are first converted into numerical vectors using word embeddings (learned representations capturing meaning) and positional encodings (to indicate word order, since Transformers don’t process sequentially). These encoded vectors are the input representations for self-attention.
• Vector Decoding: The decoder generates output sequences (e.g., translations) by transforming context-aware vectors into probability distributions over the vocabulary, selecting the most likely words.

Example: Processing “The cat runs”

Let’s illustrate with the sentence “The cat runs”, focusing on “cat” to show how self-attention, encoding, and decoding work.

Step 1: Vector Encoding

Each word is encoded into a vector:

1. Word Embeddings: Words are mapped to vectors (e.g., 512 dimensions in the Transformer, but we’ll use 2D for simplicity):
   • “The” → [1, 0]
   • “Cat” → [0, 1]
   • “Runs” → [1, 1]
2. Positional Encoding: Positional encodings are added to indicate order (e.g., “The” is first, “Cat” is second). For simplicity, assume these are included in the vectors.

These encoded vectors are fed into self-attention.

Step 2: Scaled Dot-Product Attention

Self-attention computes a new representation for “cat” by weighing all words’ vectors based on relevance:

1. Queries, Keys, Values: Each word’s vector is transformed into query (Q), key (K), and value (V) vectors. For simplicity, assume Q = K = V = input vectors.
2. Attention Scores: Compute dot products between “cat’s” query ([0, 1]) and each key:
   • “Cat” with “The”: [0, 1] · [1, 0] = 0
   • “Cat” with “Cat”: [0, 1] · [0, 1] = 1
   • “Cat” with “Runs”: [0, 1] · [1, 1] = 1
3. Scaling: Divide by √2 (≈1.414) to stabilize gradients: [0, 0.707, 0.707]
4. Softmax: Convert to weights: [0.198, 0.401, 0.401]
5. Weighted Sum: Compute the new representation for “cat”:
   • (0.198 × [1, 0]) + (0.401 × [0, 1]) + (0.401 × [1, 1]) = [0.599, 0.802]

This vector [0.599, 0.802] captures the context of “cat” in relation to “runs” and “the.”

Step 3: Matrix Operations

In practice, self-attention is computed for all words simultaneously:

• Stack input vectors into a matrix X.
• Compute Q = XW_Q, K = XW_K, V = XW_V (where W_Q, W_K, W_V are learned).
• Calculate:
  [ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V ]
  This enables parallel processing, making Transformers scalable.

Step 4: Decoding

In the Transformer, the encoder produces context-aware vectors for the input. The decoder generates the output (e.g., “Le chat court” for French translation) by:

• Encoding its input (partially generated text) into vectors.
• Using masked self-attention to focus only on prior words.
• Attending to the encoder’s vectors to align with the input.
• Decoding the final vector into a word via a linear layer and softmax.

The Journey to GPT: Building on the Transformer

The Transformer’s innovations—self-attention, parallel processing, and scalable vector encoding/decoding—set the stage for GPT, developed by OpenAI in 2018. Here’s how the journey unfolded:

1. Adopting the Decoder-Only Architecture:
   • The Transformer has both an encoder (for input) and a decoder (for output), ideal for tasks like translation. GPT, however, focuses on generative tasks (e.g., writing text), so it uses only the decoder, optimized for autoregressive generation—predicting the next word based on prior context.
   • In our example, GPT would encode “The cat,” use self-attention to weigh their relevance, and decode a vector to predict “runs.” This mirrors the Transformer’s decoder but skips the encoder, as GPT generates text without a fixed input sentence.
2. Leveraging Self-Attention for Context:
   • GPT relies on the Transformer’s masked self-attention, ensuring it only attends to previous words (like the decoder’s masked attention). This allows GPT to generate coherent text by focusing on relevant context, as seen in our example where “cat” attends strongly to “runs.”
3. Pre-training and Fine-tuning Paradigm:
   • The Transformer’s ability to scale inspired GPT’s training approach. GPT is pre-trained on vast text datasets (e.g., books, websites) to learn general language patterns, encoding words into rich vectors and refining them via self-attention. It’s then fine-tuned for specific tasks (e.g., answering questions).
   • This paradigm, enabled by the Transformer’s efficiency, allows GPT to generalize across diverse tasks, from writing stories to coding.
4. Scaling to Billions of Parameters:
   • The Transformer’s parallelizable design (via matrix operations) allowed GPT to scale to massive sizes (e.g., GPT-3’s 175 billion parameters). This scalability, rooted in the Transformer’s architecture, enables GPT to handle long contexts and generate human-like text.
5. From Translation to Generation:
   • The Transformer was designed for tasks like translation, where the encoder processes the input and the decoder generates the output. GPT extended this to open-ended generation, using the decoder to produce text without a predefined input sequence. This shift was inspired by the Transformer’s ability to capture context via self-attention, making GPT versatile for chatbots, writing, and more.

The Legacy: GPT and Beyond

The journey from RNNs to GPT is a story of overcoming limitations. RNNs struggled with sequential processing and long-range dependencies, prompting Google’s Transformer to introduce self-attention and parallel computation. By encoding words into vectors, using self-attention to create context-aware representations, and decoding these into text, the Transformer revolutionized NLP. OpenAI built on this foundation, adapting the decoder for generative tasks, scaling it with massive datasets, and creating GPT—a model that powers everything from virtual assistants to automated content creation.

For a deeper dive, check out the original paper here or explore GPT’s evolution through OpenAI’s publications. The Transformer’s legacy lives on in GPT, proving that attention truly is all you need to unlock AI’s potential.