Transformer

A detailed, step-by-step explanation of how Transformer inference works, covering encoder outputs, autoregressive decoding, masked self-attention, cross-attention, and token-by-token generation with clear mathematical intuition.

A complete, step-by-step explanation of the Transformer decoder architecture, covering masked self-attention, cross-attention, feed-forward networks, and the final softmax output using an English-to-Hindi translation example.

Cross attention is a key mechanism in transformer encoder–decoder models that allows the decoder to focus on relevant parts of the input sequence. This guide explains cross attention step by step, compares it with self-attention, and shows how output representations are formed using input context.

Transformer decoders behave autoregressively during inference but allow parallel computation during training. This post explains why naive parallel self-attention causes data leakage and how masked self-attention solves this problem while preserving autoregressive behavior.

A clear, step-by-step explanation of the Transformer encoder architecture, covering tokenization, positional encoding, self-attention, feed-forward networks, residual connections, and why multiple encoder blocks are used.