Module 16 — Vision Language Models (VLMs)
Introduction
Vision Language Models (VLMs) are multimodal AI models that can understand both images and text simultaneously. They combine Vision Transformers (ViT) or other vision encoders with Large Language Models (LLMs) to perform reasoning across multiple modalities.
Unlike traditional computer vision models that only classify images, VLMs can:
- Describe images
- Answer questions about images
- Read documents
- Understand charts and tables
- Perform OCR
- Analyze screenshots
- Understand diagrams
- Follow visual instructions
Modern AI assistants are largely built on Vision Language Models.
Applications include:
- Image Captioning
- Visual Question Answering (VQA)
- Document AI
- OCR
- Medical Imaging
- Robotics
- Autonomous Driving
- Multimodal Search
In this module, you'll learn:
- CLIP
- BLIP
- BLIP-2
- Flamingo
- LLaVA
- Qwen2-VL
- InternVL
- Kosmos
- Florence
- PaddleOCR-VL
- Image Captioning
- Visual Question Answering (VQA)
Vision Language Model Architecture
A Vision Language Model combines a vision encoder with a language model.
1 Image 2 │ 3 Vision Encoder (ViT) 4 │ 5 Visual Embeddings 6 │ 7 Multimodal Projector 8 │ 9 Language Model 10 │ 11 Text Generation 12 │ 13 Caption / Answer
The vision encoder extracts image features, which are projected into the language model's embedding space before text generation.
1. CLIP (Contrastive Language–Image Pretraining)
What is CLIP?
CLIP learns a shared embedding space for images and text. Instead of generating text, it measures how well an image and a text description match.
Architecture
1Image 2 │ 3Vision Encoder 4 │ 5Image Embedding 6 7Text 8 │ 9Text Encoder 10 │ 11Text Embedding 12 13↓ 14 15Similarity Score
Applications
- Zero-shot image classification
- Image retrieval
- Cross-modal search
- Image-text similarity
Load CLIP
1from transformers import CLIPProcessor 2from transformers import CLIPModel 3 4model_name = "openai/clip-vit-base-patch32" 5 6processor = CLIPProcessor.from_pretrained(model_name) 7 8model = CLIPModel.from_pretrained(model_name)
2. BLIP (Bootstrapping Language-Image Pretraining)
What is BLIP?
BLIP is designed for both understanding and generating text from images.
Tasks
- Image Captioning
- Visual Question Answering
- Image Retrieval
Architecture
1Image 2 3↓ 4 5Vision Encoder 6 7↓ 8 9Cross Attention 10 11↓ 12 13Language Decoder 14 15↓ 16 17Caption
Load BLIP
1from transformers import BlipProcessor 2from transformers import BlipForConditionalGeneration 3 4processor = BlipProcessor.from_pretrained( 5 "Salesforce/blip-image-captioning-base" 6) 7 8model = BlipForConditionalGeneration.from_pretrained( 9 "Salesforce/blip-image-captioning-base" 10)
3. BLIP-2
What is BLIP-2?
BLIP-2 connects a frozen vision encoder with a frozen LLM using a lightweight Q-Former.
Architecture
1Image 2 3↓ 4 5Vision Encoder 6 7↓ 8 9Q-Former 10 11↓ 12 13LLM 14 15↓ 16 17Generated Text
Advantages
- Efficient training
- Strong multimodal reasoning
- Supports powerful LLM backends
Load BLIP-2
1from transformers import Blip2Processor 2from transformers import Blip2ForConditionalGeneration 3 4processor = Blip2Processor.from_pretrained( 5 "Salesforce/blip2-opt-2.7b" 6) 7 8model = Blip2ForConditionalGeneration.from_pretrained( 9 "Salesforce/blip2-opt-2.7b" 10)
4. Flamingo
What is Flamingo?
Flamingo is a multimodal model designed for few-shot learning.
Features
- Frozen Vision Encoder
- Frozen Language Model
- Cross-attention layers
- Few-shot multimodal prompting
Applications
- Visual reasoning
- Image dialogue
- VQA
Architecture
1Image 2 3↓ 4 5Vision Encoder 6 7↓ 8 9Cross Attention 10 11↓ 12 13LLM 14 15↓ 16 17Response
5. LLaVA (Large Language and Vision Assistant)
What is LLaVA?
LLaVA combines a CLIP vision encoder with a LLaMA language model.
Pipeline
1Image 2 3↓ 4 5CLIP Encoder 6 7↓ 8 9Projection Layer 10 11↓ 12 13LLaMA 14 15↓ 16 17Answer
Capabilities
- Chat with images
- OCR
- Diagram understanding
- Chart analysis
- Visual reasoning
Load LLaVA
1from transformers import LlavaProcessor 2from transformers import LlavaForConditionalGeneration 3 4processor = LlavaProcessor.from_pretrained( 5 "llava-hf/llava-1.5-7b-hf" 6) 7 8model = LlavaForConditionalGeneration.from_pretrained( 9 "llava-hf/llava-1.5-7b-hf" 10)
6. Qwen2-VL
What is Qwen2-VL?
Qwen2-VL is Alibaba's multimodal language model.
Features
- Image understanding
- OCR
- Video understanding
- Long-context reasoning
- Multilingual support
Applications
- Document AI
- Visual QA
- Charts
- Screenshots
- Image reasoning
Load Qwen2-VL
1from transformers import ( 2 AutoProcessor, 3 Qwen2VLForConditionalGeneration 4) 5 6processor = AutoProcessor.from_pretrained( 7 "Qwen/Qwen2-VL-2B-Instruct" 8) 9 10model = Qwen2VLForConditionalGeneration.from_pretrained( 11 "Qwen/Qwen2-VL-2B-Instruct" 12)
7. InternVL
What is InternVL?
InternVL is an open-source family of multimodal large language models optimized for strong visual understanding.
Features
- High-resolution image support
- OCR
- Chart reasoning
- Mathematical reasoning
- Multimodal conversation
Applications
- Education
- Scientific diagrams
- Document analysis
- Vision-based assistants
8. Kosmos
What is Kosmos?
Kosmos is Microsoft's multimodal Transformer family that integrates language, images, and other modalities into a unified representation.
Capabilities
- Visual grounding
- OCR
- Image captioning
- Object localization
- Multimodal reasoning
Architecture
1Image + Text 2 3↓ 4 5Shared Multimodal Encoder 6 7↓ 8 9Language Decoder 10 11↓ 12 13Generated Response
9. Florence
What is Florence?
Florence is Microsoft's vision foundation model family for general computer vision tasks.
Supported Tasks
- Image captioning
- OCR
- Object detection
- Dense region captioning
- Segmentation
- Visual grounding
Advantages
- Multi-task learning
- Unified architecture
- Flexible prompting
Load Florence-2
1from transformers import ( 2 AutoProcessor, 3 AutoModelForCausalLM 4) 5 6processor = AutoProcessor.from_pretrained( 7 "microsoft/Florence-2-base" 8) 9 10model = AutoModelForCausalLM.from_pretrained( 11 "microsoft/Florence-2-base" 12)
10. PaddleOCR-VL
What is PaddleOCR-VL?
PaddleOCR-VL is a Vision-Language Model specialized for document understanding. It combines visual encoding with language modeling to process complex documents.
Capabilities
- Optical Character Recognition (OCR)
- Table recognition
- Formula recognition
- Layout analysis
- Chart understanding
- Multilingual document parsing
Applications
- PDF understanding
- Invoice processing
- Forms
- Scientific papers
- Receipts
- Business documents
Architecture
1Document Image 2 3↓ 4 5Vision Encoder 6 7↓ 8 9Layout Analysis 10 11↓ 12 13Language Decoder 14 15↓ 16 17Structured Output
Unlike traditional OCR systems that only recognize text, PaddleOCR-VL understands the semantic structure of documents.
Comparison of Vision Language Models
| Model | Primary Capability | Vision Encoder | Language Model |
|---|---|---|---|
| CLIP | Image-text matching | ViT | Text Encoder |
| BLIP | Captioning & VQA | ViT | Language Decoder |
| BLIP-2 | Efficient multimodal generation | ViT | Frozen LLM |
| Flamingo | Few-shot multimodal learning | ViT | Frozen LLM |
| LLaVA | Visual chat | CLIP | LLaMA |
| Qwen2-VL | General multimodal reasoning | ViT | Qwen |
| InternVL | High-resolution visual understanding | ViT | InternLM |
| Kosmos | Unified multimodal learning | Vision Encoder | Transformer Decoder |
| Florence | General vision foundation model | Vision Encoder | Language Decoder |
| PaddleOCR-VL | Document understanding | Vision Encoder | Language Model |
Practice 1 — Image Captioning
Generate an image caption using BLIP.
1from PIL import Image 2from transformers import ( 3 BlipProcessor, 4 BlipForConditionalGeneration 5) 6 7image = Image.open("cat.jpg").convert("RGB") 8 9processor = BlipProcessor.from_pretrained( 10 "Salesforce/blip-image-captioning-base" 11) 12 13model = BlipForConditionalGeneration.from_pretrained( 14 "Salesforce/blip-image-captioning-base" 15) 16 17inputs = processor( 18 images=image, 19 return_tensors="pt" 20) 21 22output = model.generate(**inputs) 23 24caption = processor.decode( 25 output[0], 26 skip_special_tokens=True 27) 28 29print(caption)
What You'll Learn
- Load a pretrained image captioning model
- Convert images into model inputs
- Generate natural language captions
Practice 2 — Visual Question Answering (VQA)
Answer questions about an image using BLIP.
1from PIL import Image 2from transformers import ( 3 BlipProcessor, 4 BlipForQuestionAnswering 5) 6 7image = Image.open("street.jpg").convert("RGB") 8 9processor = BlipProcessor.from_pretrained( 10 "Salesforce/blip-vqa-base" 11) 12 13model = BlipForQuestionAnswering.from_pretrained( 14 "Salesforce/blip-vqa-base" 15) 16 17question = "How many cars are visible?" 18 19inputs = processor( 20 image, 21 question, 22 return_tensors="pt" 23) 24 25outputs = model.generate(**inputs) 26 27answer = processor.decode( 28 outputs[0], 29 skip_special_tokens=True 30) 31 32print(answer)
What You'll Learn
- Ask natural language questions about images
- Use a pretrained VQA model
- Generate answers from visual content
Mini Project — Image Chat Assistant
1from PIL import Image 2from transformers import ( 3 LlavaProcessor, 4 LlavaForConditionalGeneration 5) 6 7model_name = "llava-hf/llava-1.5-7b-hf" 8 9processor = LlavaProcessor.from_pretrained(model_name) 10 11model = LlavaForConditionalGeneration.from_pretrained(model_name) 12 13image = Image.open("example.jpg").convert("RGB") 14 15prompt = "Describe this image in detail." 16 17inputs = processor( 18 images=image, 19 text=prompt, 20 return_tensors="pt" 21) 22 23outputs = model.generate( 24 **inputs, 25 max_new_tokens=100 26) 27 28response = processor.decode( 29 outputs[0], 30 skip_special_tokens=True 31) 32 33print(response)
What You'll Learn
- Build a multimodal chatbot
- Combine image and text inputs
- Generate detailed visual descriptions
- Understand the end-to-end Vision-Language pipeline
Choosing the Right Vision-Language Model
| Task | Recommended Model |
|---|---|
| Image-Text Similarity | CLIP |
| Image Captioning | BLIP, BLIP-2 |
| Visual Question Answering | BLIP, LLaVA |
| Visual Chat | LLaVA, Qwen2-VL |
| Document Understanding | PaddleOCR-VL |
| OCR & Layout Analysis | PaddleOCR-VL |
| General Vision Foundation Tasks | Florence |
| High-Resolution Image Reasoning | InternVL |
Module Summary
After completing this module, you will be able to:
- Explain how Vision Language Models integrate vision encoders with language models.
- Understand the architectures and use cases of CLIP, BLIP, BLIP-2, Flamingo, LLaVA, Qwen2-VL, InternVL, Kosmos, Florence, and PaddleOCR-VL.
- Distinguish between contrastive models (CLIP) and generative multimodal models (BLIP, LLaVA, Qwen2-VL).
- Build image captioning systems using BLIP.
- Develop Visual Question Answering (VQA) applications.
- Understand document AI workflows using PaddleOCR-VL.
- Build multimodal assistants capable of reasoning over images and text.
- Select the appropriate Vision-Language Model for image understanding, OCR, document analysis, or conversational AI.
Next Module: Module 17 – Multimodal LLM Applications, where you'll build end-to-end projects including document understanding, OCR pipelines, image chatbots, visual RAG, chart understanding, PDF question answering, and multimodal AI agents.