Fine-tuning
In brief
Fine-tuning means continuing to train an already pre-trained model on a smaller dataset to adapt it to a specific task, style or domain.
Definition
Fine-tuning takes a model already trained on a vast corpus and continues its training on more targeted data. The model keeps its general abilities while adapting to a specific need: editorial tone, industry vocabulary, strict output format, classification task.
Variants
- Instruction tuning: training the model to follow instructions from examples (instruction, expected answer).
- Reinforcement learning from human feedback (RLHF): evaluators compare answers, and the model is adjusted to favour the ones they prefer. The paper by Ouyang et al. describes how it is applied to language models.
- Parameter-efficient methods: instead of changing all the model’s weights, only a small number of additional parameters are trained. LoRA is the best-known example and greatly reduces the cost of fine-tuning.
Fine-tuning, RAG or prompt?
- The prompt is the simplest and cheapest option: try it first.
- RAG is suited to bringing in up-to-date or private knowledge, with verifiable sources.
- Fine-tuning is suited to durably changing behaviour, style or format, or specialising a smaller model on a precise task.
These approaches are not mutually exclusive and can be combined.
Limits
It requires enough good-quality data, computing costs and rigorous evaluation. Poorly done fine-tuning can degrade the model’s general abilities.
Spotted a mistake or have an idea about this entry? Let us know