Multimodal AI: Models That See, Hear and Speak

Multimodal models understand more than one kind of input: text with images, audio or video. This guide shows the core idea, a shared space where a photo and a sentence can be compared, in a small CLIP demo, then explains how today's vision-language assistants are built, what they cost and where they fail.

GloryTecks Team

Training & Placement Faculty

6 May 2026 7 min read

Last updated: 3 October 2026

Share:

People do not experience the world one sense at a time. We read a chart and its caption together, hear a question while looking at a screen, describe a photo in words. Multimodal AI models aim at the same thing: understanding and producing more than one kind of data, usually text plus images, audio or video.

The idea that makes this possible is simpler than it sounds. If a picture and a sentence can be turned into the same kind of object, they can be compared.

One space for pictures and words

CLIP, published by OpenAI in 2021, made that idea work at scale. It was trained on 400 million image and caption pairs from the internet with one task: given a batch of images and captions, pick out which caption goes with which image. To do that well, the model has to place each image and its description close together in a shared space of numbers (Radford et al.: CLIP).

Once trained, anything you can describe in words becomes a label you can test against a picture, with no further training. That is zero-shot classification, and it is easy to try.

A demonstration

The script loads four photographs that ship with scikit-image, two in the public domain and two released under CC0 by their photographers (scikit-image: data). It turns them, and six short descriptions, into vectors with a small CLIP model. It then labels each picture and searches the pictures with sentences that name none of them.

python
import torch
from skimage import data
from transformers import CLIPModel, CLIPProcessor

# Four public-domain or CC0 photographs that ship with scikit-image
images = {"astronaut": data.astronaut(), "cat": data.chelsea(), "coffee": data.coffee(), "rocket": data.rocket()}

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32", use_fast=False)

labels = ["a photo of an astronaut", "a photo of a cat", "a cup of coffee", "a rocket launch",
          "a bowl of soup", "a photo of a dog"]

with torch.no_grad():
    image_vecs = model.get_image_features(**processor(images=list(images.values()), return_tensors="pt"))
    text_vecs = model.get_text_features(**processor(text=labels, return_tensors="pt", padding=True))
image_vecs = image_vecs / image_vecs.norm(dim=-1, keepdim=True)
text_vecs = text_vecs / text_vecs.norm(dim=-1, keepdim=True)
print(f"images and sentences both become vectors of {image_vecs.shape[1]} numbers in one shared space")

# 1. Zero-shot classification: which description is closest to each picture?
probs = (100 * image_vecs @ text_vecs.T).softmax(dim=-1)
print("\nzero-shot labels (no training on these images or labels)")
for name, row in zip(images, probs):
    best = row.argmax().item()
    print(f"   {name:<10} -> {labels[best]:<26} {row[best]:.1%}")

# 2. Search pictures with a sentence that names none of them
print("\nsearch by meaning")
for query in ["something hot to drink", "travelling to space", "a pet resting at home"]:
    with torch.no_grad():
        q = model.get_text_features(**processor(text=[query], return_tensors="pt", padding=True))
    q = q / q.norm(dim=-1, keepdim=True)
    scores = (image_vecs @ q.T).squeeze(1)
    ranked = sorted(zip(images, scores.tolist()), key=lambda x: -x[1])
    print(f"   {query!r:<26} -> " + ", ".join(f"{n} {s:.3f}" for n, s in ranked[:3]))
text
images and sentences both become vectors of 512 numbers in one shared space

zero-shot labels (no training on these images or labels)
   astronaut  -> a photo of an astronaut    100.0%
   cat        -> a photo of a cat           99.3%
   coffee     -> a cup of coffee            99.8%
   rocket     -> a rocket launch            95.6%

search by meaning
   'something hot to drink'   -> coffee 0.267, cat 0.191, rocket 0.154
   'travelling to space'      -> rocket 0.249, astronaut 0.234, cat 0.185
   'a pet resting at home'    -> cat 0.237, rocket 0.181, coffee 0.179

What the output shows

  • Pictures and sentences became the same kind of thing: vectors of 512 numbers, directly comparable.
  • Every photo got the right label, with confidence from 95.6% to almost 100%, although the model was never trained on these images or these labels. The rocket was least certain, perhaps because the launch photo shows mostly smoke and sky.
  • Search worked by meaning. "Something hot to drink" found the coffee, "travelling to space" ranked the rocket and the astronaut first and second, and "a pet resting at home" found the cat, though none of those words appear in the labels.
  • The scores are relative. Similarities of about 0.25 for good matches and 0.18 for poor ones are typical for CLIP. What matters is the ranking, not the absolute number.

The same principle extends to audio: speech or sound encoders map recordings into spaces that line up with text. See speech-to-text and text-to-speech. And it is the image version of the text embeddings behind semantic search. See embeddings explained.

Read the model card first

This CLIP model's card is unusually direct. Its intended users are AI researchers, any deployed use case, commercial or not, is described as out of scope for now, surveillance and facial recognition are always out of scope, and use should be limited to English (Hugging Face: CLIP model card). The demonstration above is the kind of research and teaching use the card describes. For a product, choose a model whose card and licence allow it, and test it on your own images.

From matching to conversation

Matching is not the same as describing. Assistants that answer questions about an image, read a chart or summarise a screenshot are vision-language models. A common design connects a vision encoder, often a CLIP-style one, to a large language model through a small adapter. The image becomes a sequence of tokens the language model can read alongside the text. LLaVA, an influential open example, was built this way and then trained on instruction-following examples about images (Liu et al.: LLaVA).

Hosted models expose this through their APIs. Anthropic's documentation for Claude gives a concrete picture of the practicalities (Anthropic: vision):

Practical pointWhat the documentation says
FormatsJPEG, PNG, GIF and WebP; animations use the first frame only
SizeUp to 8000 by 8000 pixels per image, with stricter limits when a request has many images
CostImages are read in 28 by 28 pixel patches, each counted as a visual token
Known limitsDoes not identify people; may err on small, rotated or low-quality images; approximate counts and positions; cannot tell whether an image is AI-generated

Those limits are typical of the category, and worth designing around. Ask for counts and positions only when approximations will do, keep images large enough to read, and never use a model's opinion as proof that an image is real or fake.

What multimodal models are good for

  • Search across photos, products or documents with natural language.
  • Document understanding: forms, invoices, charts and screenshots.
  • Accessibility: describing images for people who cannot see them.
  • Voice interfaces that listen and speak.
  • Quality checks that compare images against written criteria, with a person reviewing the decisions that matter.

For generating images rather than understanding them, see diffusion models explained.

Common mistakes

  • Trusting absolute similarity scores instead of rankings.
  • Deploying a research model whose card rules deployment out.
  • Asking for exact counts or positions and treating the answer as precise.
  • Sending tiny or heavily compressed images and blaming the model.
  • Ignoring cost: large images consume many tokens per request.
  • Using image models to identify people or judge authenticity.

What is multimodal AI?

AI that works with more than one kind of data, such as text with images, audio or video. Models map the different kinds of input into representations they can compare or reason over together, so they can, for example, answer questions about a picture.

What is CLIP?

A model from OpenAI trained on 400 million image and caption pairs to place images and matching text close together in a shared space. It can label images with any descriptions you supply, without extra training, and search images by meaning.

How do vision-language models like those in ChatGPT or Claude see images?

An image encoder turns the picture into a sequence of tokens that the language model reads alongside the text. Claude, for example, reads images in 28 by 28 pixel patches, each counted as a visual token.

What are the limitations of multimodal models?

They can misread small or low-quality images, give approximate counts and positions, invent details, and cannot reliably tell whether an image is AI-generated. Some models' terms also restrict uses such as identifying people.

Multimodal models are a module of the Generative AI course.

Sources

GloryTecks Team

Training & Placement Faculty

The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.

Want to master Generative AI?

Learn Generative AI at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.

Explore course