# Transformer architecture > Weighting relationships within a sequence - Date: 2017 - Era: Modern age - Domain: [Electricity & Electronics](https://lastcodex.org/en/domains/electricity/) - Rebuild stage: V (Electric age) - Difficulty: 5/5 - URL: https://lastcodex.org/en/inventions/transformer-network/ The Transformer is a neural network architecture published in 2017 by Ashish Vaswani and his coauthors. It processes sequences through attention, numerical transformations and position information; the original model has an encoder and a decoder, studied particularly for machine translation. ![Transformer architecture · Weighting relationships within a sequence](https://lastcodex.org/_astro/plate.2HFQaqM0_2jG6ls.webp) ## Legend 1. Input representations and positions 2. Self-attention encoder 3. Previous outputs and positions 4. Masked self-attention decoder 5. Attention to encoder outputs 6. Projection and output probabilities ## What it is A Transformer turns a sequence of numerical representations into another sequence. For text, the units being processed, called tokens, are assigned vectors: lists of numbers that the network transforms. In the translation architecture published in 2017, an encoder processes the input and a decoder builds the output from elements already available. The network learns parameters from training examples. Once these parameters are fixed, an input determines a computation; text generation also depends on how outputs are selected. Neural networks, machine translation and attention mechanisms already had their own histories when this combination appeared. Its contribution concerns a specific architecture for processing sequences. ## How it works ![Frontal diagram of a query comparing two keys and then combining their associated values](https://lastcodex.org/_astro/v1.BfUbmeAM_1XWpFb.webp) _In each pair, the key is on the left and the value on the right. The weighted values combine at the output._ Attention forms queries, keys and values from representations. The dot product of a query with each key gives a score, divided by the square root of the key dimension. Softmax exponentiates these scores and divides them by their sum. The resulting weights are used to add weighted values. A query can thereby gather information from several positions rather than retrieving a single item unchanged. In self-attention, queries, keys and values come from the same sequence through separate learned projections. Multiple heads operate on different projections; their outputs are joined and transformed. The encoder relates input positions to each other. The decoder masks future output positions and also consults the encoder's representations through a separate attention operation. The architecture also includes networks applied at each position, residual connections that add a sublayer's input to its result, and normalization. Position information supplements the vectors because content comparisons alone do not supply sequence order. The base model describes six encoder layers, six decoder layers and eight heads within its multi-head attention sublayers. In encoder–decoder attention, queries come from the decoder, while keys and values come from the encoder output. In the original model, layer normalisation follows each residual addition. These operations repeat at each layer, whose learned weights can differ from those of the other layers. During training, the known elements of the target output are shifted before entering the decoder. The mask prevents each position from using future elements even when several positions are computed together. During generation, the decoder receives the elements already produced and computes probabilities for the next element in the sequence. ## How to do it A [computer](/en/inventions/computer/) and a calculator are enough to explore one attention operation. Choose a key dimension of 1, a query of 1, two keys `ln(3)` and 0, and two values 2 and 6. Here `ln` means the natural logarithm. These numbers are chosen for the calculation; no training has taken place. ![A three-by-three square matrix with the cells above its diagonal hatched](https://lastcodex.org/_astro/v2.DjNEX4xZ_ZTeg0j.webp) _A causal mask blocks future positions above the diagonal: rows are queries and columns are keys._ The scores are `ln(3)` and 0, since the square root of 1 is 1. Their exponentials are 3 and 1, summing to 4. The weights are therefore 3/4 and 1/4. The output is `(3/4 × 2) + (1/4 × 6) = 3`. Giving one value the larger weight does not automatically remove the other contribution. Repeat the calculation while excluding the second key. Set its score to negative infinity before applying softmax: its exponential becomes 0. The weights become 1 and 0, and the output becomes 2. In a sequence, this rule prevents a position from consulting later elements. A triangular mask keeps the current and earlier positions accessible. Training a complete network requires representations, all the layers, data, a learning objective and parameter updates. ## History and evolution In 2014, Bahdanau, Cho and Bengio proposed learned alignment for neural translation; the work was presented at ICLR in 2015. Their decoder could weight different parts of a source sentence. Self-attention was also explored before the Transformer, as the 2017 paper acknowledges. On 12 June 2017, Vaswani and seven coauthors submitted Attention Is All You Need to arXiv. They built an encoder and decoder without recurrence or convolution for relating positions. Computation of representations allowed greater parallelism, while autoregressive generation still produced successive output elements in order. Later developments use encoders, decoders or both for other tasks. These variants change applications and training methods. The architecture published in 2017 remains the starting point for the Transformer family. ## Key facts - Initial arXiv publication: 2017 - Original encoder and decoder layers: 6 + 6 - Attention heads in the base model: 8 ## Requires - [Computer](https://lastcodex.org/en/inventions/computer/): A machine with its program in memory (1948) ## Sources - [Attention Is All You Need, version 1, 12 juin 2017](https://arxiv.org/pdf/1706.03762v1), Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin ; arXiv - [The Transformer Architecture, Dive into Deep Learning, §11.7](https://d2l.ai/chapter_attention-mechanisms-and-transformers/transformer.html), Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola ; Dive into Deep Learning - [Attention Scoring Functions, Dive into Deep Learning, §11.3](https://d2l.ai/chapter_attention-mechanisms-and-transformers/attention-scoring-functions.html), Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola ; Dive into Deep Learning - [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473), Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio ; arXiv - [Attention Is All You Need, notice et historique des versions](https://arxiv.org/abs/1706.03762), arXiv, métadonnées de la contribution Vaswani et al. Fact-checked on 2026-11-01. --- The Last Codex · https://lastcodex.org/en/inventions/transformer-network/ · © The Last Codex. Texts, plates and figures: CC BY-NC-ND 4.0.