Multimodal models: why text, image and audio are converging

Direct answer: multimodal models are artificial intelligence systems that process, and in some cases generate, more than one type of data, such as text, image, audio and video, in a single system. They are converging because similar techniques serve several types of data and because combining information in different forms makes tasks more useful. Features vary between products and change fast.

Three points to understand

1. Why they converge. Neural network architectures used for text have proved adaptable to images and audio, when those data are converted into comparable numerical representations. Training the system with pairs, such as image and caption, or speech and transcript, helps connect the modalities.
2. What this allows. Describing a photo, answering questions about a chart, transcribing and summarizing audio, reading a scanned document, talking by voice and generating images from text. For the user, interaction becomes more natural, since you can speak, show and write with the same tool.
3. Where they still fail. They can invent details that do not exist, err in reading charts and tables, lose subtle information from images and audio and reflect biases in the data they were trained on. They also raise privacy questions, since they deal with photos, voices and documents.

An example

A person photographs a product label in another language and asks the assistant to translate and explain the ingredients. The system reads the image, translates the text and answers. If the photo is blurry, it may err or invent passages, so it is worth checking the original label. This is an illustrative example.

Practical care

• Check the answers on important matters, such as health, money and documents.
• Avoid sending sensitive data (personal documents, intimate photos, third-party information) without understanding the service's privacy rules.
• Understand the limits of each product, which vary between versions.
• Check copyright and consent when generating or sharing images and voices of other people.

Next reading

Also see the articles on prompt injection and on how to check answers from AI search engines, which deal with practical risks in using these tools.