Multimodal AI: Models That See, Hear and Speak
Multimodal models understand more than one kind of input: text with images, audio or video. This guide shows the core idea, a shared space where a photo and a sentence can be compared, in a small CLIP demo, then explains how today's vision-language assistants are built, what they cost and where they fail.
GloryTecks Team
Training & Placement Faculty
Last updated: 3 October 2026
People do not experience the world one sense at a time. We read a chart and its caption together, hear a question while looking at a screen, describe a photo in words. Multimodal AI models aim at the same thing: understanding and producing more than one kind of data, usually text plus images, audio or video.
The idea that makes this possible is simpler than it sounds. If a picture and a sentence can be turned into the same kind of object, they can be compared.
One space for pictures and words
CLIP, published by OpenAI in 2021, made that idea work at scale. It was trained on 400 million image and caption pairs from the internet with one task: given a batch of images and captions, pick out which caption goes with which image. To do that well, the model has to place each image and its description close together in a shared space of numbers (Radford et al.: CLIP).
Once trained, anything you can describe in words becomes a label you can test against a picture, with no further training. That is zero-shot classification, and it is easy to try.
A demonstration
The script loads four photographs that ship with scikit-image, two in the public domain and two released under CC0 by their photographers (scikit-image: data). It turns them, and six short descriptions, into vectors with a small CLIP model. It then labels each picture and searches the pictures with sentences that name none of them.
import torch
from skimage import data
from transformers import CLIPModel, CLIPProcessor
# Four public-domain or CC0 photographs that ship with scikit-image
images = {"astronaut": data.astronaut(), "cat": data.chelsea(), "coffee": data.coffee(), "rocket": data.rocket()}
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32", use_fast=False)
labels = ["a photo of an astronaut", "a photo of a cat", "a cup of coffee", "a rocket launch",
"a bowl of soup", "a photo of a dog"]
with torch.no_grad():
image_vecs = model.get_image_features(**processor(images=list(images.values()), return_tensors="pt"))
text_vecs = model.get_text_features(**processor(text=labels, return_tensors="pt", padding=True))
image_vecs = image_vecs / image_vecs.norm(dim=-1, keepdim=True)
text_vecs = text_vecs / text_vecs.norm(dim=-1, keepdim=True)
print(f"images and sentences both become vectors of {image_vecs.shape[1]} numbers in one shared space")
# 1. Zero-shot classification: which description is closest to each picture?
probs = (100 * image_vecs @ text_vecs.T).softmax(dim=-1)
print("\nzero-shot labels (no training on these images or labels)")
for name, row in zip(images, probs):
best = row.argmax().item()
print(f" {name:<10} -> {labels[best]:<26} {row[best]:.1%}")
# 2. Search pictures with a sentence that names none of them
print("\nsearch by meaning")
for query in ["something hot to drink", "travelling to space", "a pet resting at home"]:
with torch.no_grad():
q = model.get_text_features(**processor(text=[query], return_tensors="pt", padding=True))
q = q / q.norm(dim=-1, keepdim=True)
scores = (image_vecs @ q.T).squeeze(1)
ranked = sorted(zip(images, scores.tolist()), key=lambda x: -x[1])
print(f" {query!r:<26} -> " + ", ".join(f"{n} {s:.3f}" for n, s in ranked[:3]))images and sentences both become vectors of 512 numbers in one shared space
zero-shot labels (no training on these images or labels)
astronaut -> a photo of an astronaut 100.0%
cat -> a photo of a cat 99.3%
coffee -> a cup of coffee 99.8%
rocket -> a rocket launch 95.6%
search by meaning
'something hot to drink' -> coffee 0.267, cat 0.191, rocket 0.154
'travelling to space' -> rocket 0.249, astronaut 0.234, cat 0.185
'a pet resting at home' -> cat 0.237, rocket 0.181, coffee 0.179What the output shows
- Pictures and sentences became the same kind of thing: vectors of 512 numbers, directly comparable.
- Every photo got the right label, with confidence from 95.6% to almost 100%, although the model was never trained on these images or these labels. The rocket was least certain, perhaps because the launch photo shows mostly smoke and sky.
- Search worked by meaning. "Something hot to drink" found the coffee, "travelling to space" ranked the rocket and the astronaut first and second, and "a pet resting at home" found the cat, though none of those words appear in the labels.
- The scores are relative. Similarities of about 0.25 for good matches and 0.18 for poor ones are typical for CLIP. What matters is the ranking, not the absolute number.
The same principle extends to audio: speech or sound encoders map recordings into spaces that line up with text. See speech-to-text and text-to-speech. And it is the image version of the text embeddings behind semantic search. See embeddings explained.
Read the model card first
This CLIP model's card is unusually direct. Its intended users are AI researchers, any deployed use case, commercial or not, is described as out of scope for now, surveillance and facial recognition are always out of scope, and use should be limited to English (Hugging Face: CLIP model card). The demonstration above is the kind of research and teaching use the card describes. For a product, choose a model whose card and licence allow it, and test it on your own images.
From matching to conversation
Matching is not the same as describing. Assistants that answer questions about an image, read a chart or summarise a screenshot are vision-language models. A common design connects a vision encoder, often a CLIP-style one, to a large language model through a small adapter. The image becomes a sequence of tokens the language model can read alongside the text. LLaVA, an influential open example, was built this way and then trained on instruction-following examples about images (Liu et al.: LLaVA).
Hosted models expose this through their APIs. Anthropic's documentation for Claude gives a concrete picture of the practicalities (Anthropic: vision):
| Practical point | What the documentation says |
|---|---|
| Formats | JPEG, PNG, GIF and WebP; animations use the first frame only |
| Size | Up to 8000 by 8000 pixels per image, with stricter limits when a request has many images |
| Cost | Images are read in 28 by 28 pixel patches, each counted as a visual token |
| Known limits | Does not identify people; may err on small, rotated or low-quality images; approximate counts and positions; cannot tell whether an image is AI-generated |
Those limits are typical of the category, and worth designing around. Ask for counts and positions only when approximations will do, keep images large enough to read, and never use a model's opinion as proof that an image is real or fake.
What multimodal models are good for
- Search across photos, products or documents with natural language.
- Document understanding: forms, invoices, charts and screenshots.
- Accessibility: describing images for people who cannot see them.
- Voice interfaces that listen and speak.
- Quality checks that compare images against written criteria, with a person reviewing the decisions that matter.
For generating images rather than understanding them, see diffusion models explained.
Common mistakes
- Trusting absolute similarity scores instead of rankings.
- Deploying a research model whose card rules deployment out.
- Asking for exact counts or positions and treating the answer as precise.
- Sending tiny or heavily compressed images and blaming the model.
- Ignoring cost: large images consume many tokens per request.
- Using image models to identify people or judge authenticity.
What is multimodal AI?
What is CLIP?
How do vision-language models like those in ChatGPT or Claude see images?
What are the limitations of multimodal models?
Multimodal models are a module of the Generative AI course.
Sources
- Learning Transferable Visual Models From Natural Language Supervision (arXiv) — Radford et al.
- openai/clip-vit-base-patch32 model card (Hugging Face) — OpenAI
- Visual Instruction Tuning (arXiv) — Liu et al.
- Vision (Claude Platform Docs) — Anthropic
- skimage.data (scikit-image documentation) — scikit-image
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Generative AI?
Learn Generative AI at GloryTecks in Hyderabad, with trainers who have industry experience, real-time projects and placement assistance.
Related articles
Amazon Bedrock vs Microsoft Foundry vs Google's Agent Platform in 2027
AWS, Microsoft and Google rebuilt their generative AI platforms around agents in 2025-26: Bedrock with AgentCore, Microsoft Foundry (formerly Azure AI Foundry) and Gemini Enterprise Agent Platform (formerly Vertex AI). This comparison covers models, agents, retrieval, governance, certifications and how to choose.
AI Agent Frameworks in 2027: LangGraph, Microsoft Agent Framework, OpenAI Agents SDK, Google ADK and CrewAI
A practical comparison of the five agent frameworks most teams choose between in 2027: LangChain with LangGraph, Microsoft Agent Framework, the OpenAI Agents SDK, Google's ADK and CrewAI. What each is, how they differ on state, approvals, multi-agent work and tracing, and which to learn first.
What Is Generative AI? A Complete Beginner's Guide
What generative AI is, in plain language: how language models produce text one token at a time (shown with a tiny runnable model), how chat models are trained, how image models differ, what the technology is reliable for, and why it confabulates.