Transformer Fundamentals

Diagram explaining why it is called self-attention in transformers, showing how words in a sentence (‘the cat sat on the mat’) attend to each other using query, key, and value interactions, illustrating that the sequence attends to itself.

Why Is Self-Attention Called “Self”? Understanding Attention Mechanisms from Encoder–Decoder to Transformers

This blog explains why self-attention qualifies as an attention mechanism and why the term “self” is used. By revisiting encoder–decoder attention, Luong attention, and alignment scores, we build a clear intuition for how self-attention works within a single sequence.

Aryan

Feb 28

Why Is Self-Attention Called “Self”? Understanding Attention Mechanisms from Encoder–Decoder to Transformers

© 2025 Aryan Upadhyay |