Multimodal AI
In brief
A multimodal AI understands or produces several types of data at once (text, image, sound, video): for example describing a photo, answering a question about a chart or holding a spoken conversation.
Definition
A modality is a type of data: text, image, audio, video. A multimodal model can handle several of them in the same conversation. You can send it a photo of an invoice and ask for a summary, speak to it instead of typing, or have it produce an image from a description.
How it works
Each type of data is converted into a shared numerical representation (embeddings). The model learns to connect these representations: a photo of a cat and the word "cat" end up close together. The CLIP work (2021) showed that this link could be learned from images paired with their captions; models such as Flamingo (2022) then made it possible to answer questions about images and videos from just a few examples.
Example uses
- Describe an image for a visually impaired person.
- Extract information from a scanned document, a table or a chart.
- Talk with an assistant out loud, in real time.
- Analyse a video or a screenshot to explain what it shows.
Limitations
- Image reading is still imperfect: small text, complex diagrams or details can be misread.
- A photo or recording may contain personal data (faces, voices, documents): check what you send.
- Capabilities vary widely between models: not all of them handle video or audio.
Spotted a mistake or have an idea about this entry? Let us know