LLM (large language model)

Definition

An LLM (Large Language Model) is a neural network trained on very large text corpora. Its basic training consists of predicting the next word (or more precisely the next token) in a sequence. By repeating this billions of times, the model acquires a statistical representation of language, grammar, part of the knowledge present in its data, and some reasoning patterns.

How it works

• Tokenisation: the text is split into units called tokens (words, word fragments, punctuation marks), which are converted into numbers.
• Transformer architecture: most current LLMs are based on this architecture, introduced in 2017. Its attention mechanism lets the model weigh the importance of each part of the input text when producing what comes next.
• Pre-training: the model learns to predict the next token on a massive corpus.
• Fine-tuning: an additional phase (for example learning from human feedback) makes the model more useful, safer and better at following instructions.
• Generation: in use, the model produces its answer token by token, choosing at each step among the most likely continuations.

What an LLM can do

Write and rephrase, summarise, translate, answer questions, classify or extract information, write code and hold a conversation in natural language. The most recent models also handle other modalities (images, audio) and can use external tools.

Limits to keep in mind

• Hallucinations: an LLM can produce false statements with confidence, because it generates plausible text rather than verified truth.
• Dated knowledge: its knowledge stops at the end of its training, unless it is connected to up-to-date sources.
• Bias: it reflects the biases present in its training data.
• Limited context window: it can only process a finite amount of text at once.
• Cost: training and running large models requires significant computing resources.

A little history

The paper “Attention Is All You Need” (2017) introduced the Transformer architecture. From 2018, model families such as GPT and BERT showed the value of large-scale pre-training. The GPT-3 paper (2020) showed that a very large model can perform new tasks from just a few examples given in the prompt. Since then, public access through conversational assistants has made LLMs ubiquitous.

Not to be confused with

• Generative AI: a broader category that also includes generating images, audio and video. LLMs are its text branch.
• Search engine: by default an LLM does not look things up in a document base; it generates an answer from what it has learned.