Transformer

Definition

The Transformer is a neural network architecture introduced in 2017 for machine translation. It replaced recurrent networks in most language tasks, then spread to images, audio and other modalities.

The principle: attention

To produce each output element, the model computes attention weights that indicate how important every other element of the sequence is. In self-attention, each token looks at all the other tokens in the text to build a context-aware representation. Several attention “heads” work in parallel, each able to capture a different kind of relationship.

Why it took over

• Parallelisation: unlike recurrent networks, which read text step by step, the Transformer processes the whole sequence in parallel during training, making good use of graphics processors (GPUs).
• Long-range dependencies: attention directly links words that are far apart in the text.
• Scaling: the architecture lends itself to training very large models on very large corpora.

Main limitation

The computing cost of self-attention grows sharply with the length of the text, which limits the size of the context window and has motivated many more efficient variants.