Module 25 — Efficient Transformers
Introduction
As Large Language Models (LLMs) grow from millions to hundreds of billions of parameters, efficient inference becomes essential. Running these models without optimization often requires large amounts of GPU memory, high bandwidth, and significant compute resources.
Efficient Transformers focus on reducing memory usage, improving inference speed, and enabling deployment on consumer GPUs, servers, edge devices, and cloud platforms.
Modern optimization techniques include:
- Quantization
- GPTQ
- AWQ
- GGUF
- ONNX
- TensorRT
- vLLM
- Tensor Parallelism
- Pipeline Parallelism
- Speculative Decoding
These technologies power production AI systems such as chatbots, coding assistants, search engines, document analysis tools, and enterprise AI platforms.
In this module, you'll learn:
- Quantization
- GPTQ
- AWQ
- GGUF
- ONNX
- TensorRT
- vLLM
- Tensor Parallelism
- Pipeline Parallelism
- Speculative Decoding
- Deploy a Quantized LLM
Efficient Inference Pipeline
1 Pretrained Model 2 │ 3 ▼ 4 Quantization 5 │ 6 ▼ 7 Optimized Model Format 8 (GPTQ / AWQ / GGUF / ONNX) 9 │ 10 ▼ 11 Optimized Runtime (vLLM/TensorRT) 12 │ 13 ▼ 14 Fast & Efficient Inference
Why Model Optimization?
Large models require:
- Large GPU memory
- High bandwidth
- Long inference times
- High deployment costs
Optimization techniques reduce these requirements while preserving model quality.
Benefits
- Faster inference
- Lower latency
- Lower VRAM usage
- Reduced deployment costs
- Support for consumer hardware
1. Quantization
What is Quantization?
Quantization converts model weights from high-precision data types (such as FP32 or FP16) into lower-precision formats (such as INT8 or INT4).
Pipeline
1FP32 Model 2 3↓ 4 5INT8 / INT4 6 7↓ 8 9Smaller Model 10 11↓ 12 13Faster Inference
Common Formats
| Format | Memory | Speed | Accuracy |
|---|---|---|---|
| FP32 | Highest | Slow | Highest |
| FP16 | Medium | Faster | Very High |
| BF16 | Medium | Faster | Very High |
| INT8 | Low | Fast | High |
| INT4 | Very Low | Very Fast | Moderate–High |
Advantages
- Smaller model size
- Lower memory consumption
- Faster inference
- Reduced deployment cost
Example with BitsAndBytes
1from transformers import ( 2 AutoModelForCausalLM, 3 BitsAndBytesConfig 4) 5 6bnb_config = BitsAndBytesConfig( 7 load_in_4bit=True 8) 9 10model = AutoModelForCausalLM.from_pretrained( 11 "meta-llama/Llama-3.2-1B", 12 quantization_config=bnb_config 13)
2. GPTQ
What is GPTQ?
GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization technique that compresses LLMs into low-bit representations without retraining.
Pipeline
1FP16 Model 2 3↓ 4 5GPTQ 6 7↓ 8 94-bit Model 10 11↓ 12 13Inference
Advantages
- High-quality 4-bit quantization
- No additional training required
- Significant VRAM reduction
- Widely supported by inference engines
Applications
- Local LLM deployment
- Consumer GPUs
- Edge inference
3. AWQ
What is AWQ?
AWQ (Activation-aware Weight Quantization) improves quantization quality by considering activation statistics when selecting weight scales.
Pipeline
1Weights 2 3+ 4 5Activations 6 7↓ 8 9AWQ 10 11↓ 12 13Optimized INT4 Model
Advantages
- Better accuracy than naive quantization
- Excellent performance on instruction-tuned models
- Low memory usage
- Production-ready
4. GGUF
What is GGUF?
GGUF (GPT-Generated Unified Format) is a model file format commonly used by inference engines such as llama.cpp.
Features
- Optimized for CPU inference
- Supports multiple quantization levels
- Portable model format
- Stores tokenizer and metadata
Architecture
1Model 2 3↓ 4 5GGUF Conversion 6 7↓ 8 9llama.cpp 10 11↓ 12 13Inference
Common Quantization Levels
- Q2_K
- Q3_K
- Q4_K_M
- Q5_K_M
- Q6_K
- Q8_0
Applications
- Local AI
- CPU inference
- Edge devices
- Desktop assistants
5. ONNX
What is ONNX?
Open Neural Network Exchange (ONNX) is an interoperable model format that enables deployment across different frameworks and hardware.
Pipeline
1PyTorch 2 3↓ 4 5Export 6 7↓ 8 9ONNX 10 11↓ 12 13ONNX Runtime
Advantages
- Cross-platform
- Hardware acceleration
- Optimized execution
- Broad ecosystem support
Export to ONNX
1import torch 2 3dummy_input = tokenizer( 4 "Hello", 5 return_tensors="pt" 6) 7 8torch.onnx.export( 9 model, 10 (dummy_input["input_ids"],), 11 "model.onnx" 12)
6. TensorRT
What is TensorRT?
TensorRT is NVIDIA's inference optimization framework for NVIDIA GPUs.
Features
- Kernel fusion
- Graph optimization
- Mixed precision
- INT8 optimization
- Low latency
Pipeline
1ONNX Model 2 3↓ 4 5TensorRT Engine 6 7↓ 8 9Optimized GPU Inference
Advantages
- Maximum GPU performance
- Lower latency
- Higher throughput
7. vLLM
What is vLLM?
vLLM is a high-performance inference engine designed specifically for serving Large Language Models.
Key Features
- PagedAttention
- Continuous batching
- High throughput
- Streaming generation
- OpenAI-compatible API
Architecture
1Requests 2 3↓ 4 5Continuous Batching 6 7↓ 8 9PagedAttention 10 11↓ 12 13GPU 14 15↓ 16 17Responses
Advantages
- Excellent request throughput
- Efficient KV cache management
- Production-ready LLM serving
Simple vLLM Example
1from vllm import LLM, SamplingParams 2 3llm = LLM( 4 model="meta-llama/Llama-3.2-1B" 5) 6 7params = SamplingParams( 8 temperature=0.7, 9 max_tokens=100 10) 11 12outputs = llm.generate( 13 ["Explain Transformers."], 14 params 15) 16 17print(outputs[0].outputs[0].text)
8. Tensor Parallelism
What is Tensor Parallelism?
Tensor Parallelism splits individual neural network layers across multiple GPUs.
Architecture
1Linear Layer 2 3↓ 4 5GPU 1 6 7GPU 2 8 9GPU 3 10 11↓ 12 13Combined Output
Advantages
- Supports larger models
- Efficient multi-GPU inference
- Reduces per-GPU memory usage
9. Pipeline Parallelism
What is Pipeline Parallelism?
Pipeline Parallelism distributes different model layers across different GPUs.
Pipeline
1GPU 1 2 3Layers 1–10 4 5↓ 6 7GPU 2 8 9Layers 11–20 10 11↓ 12 13GPU 3 14 15Layers 21–30
Advantages
- Scales very deep models
- Better GPU memory utilization
- Suitable for both training and inference
10. Speculative Decoding
What is Speculative Decoding?
Speculative decoding accelerates generation by using a smaller draft model to propose tokens, which are then verified by a larger target model.
Pipeline
1Prompt 2 3↓ 4 5Small Draft Model 6 7↓ 8 9Candidate Tokens 10 11↓ 12 13Large Target Model 14 15↓ 16 17Accepted Output
Advantages
- Lower latency
- Faster text generation
- Preserves output quality when verification succeeds
Comparison of Optimization Techniques
| Technique | Purpose | Primary Benefit |
|---|---|---|
| Quantization | Reduce precision | Smaller models, faster inference |
| GPTQ | Post-training quantization | Efficient 4-bit deployment |
| AWQ | Activation-aware quantization | Better accuracy at low precision |
| GGUF | Optimized model format | Efficient local CPU inference |
| ONNX | Interoperable model format | Cross-platform deployment |
| TensorRT | NVIDIA optimization | High-performance GPU inference |
| vLLM | LLM serving engine | High throughput and efficient batching |
| Tensor Parallelism | Split layers across GPUs | Larger model support |
| Pipeline Parallelism | Split model stages | Better scaling across GPUs |
| Speculative Decoding | Faster generation | Reduced latency |
Practice — Deploy a Quantized LLM
Step 1 — Load a 4-bit Quantized Model
1from transformers import ( 2 AutoTokenizer, 3 AutoModelForCausalLM, 4 BitsAndBytesConfig 5) 6 7model_name = "meta-llama/Llama-3.2-1B" 8 9tokenizer = AutoTokenizer.from_pretrained(model_name) 10 11config = BitsAndBytesConfig( 12 load_in_4bit=True 13) 14 15model = AutoModelForCausalLM.from_pretrained( 16 model_name, 17 quantization_config=config, 18 device_map="auto" 19)
Step 2 — Generate Text
1prompt = "Explain self-attention." 2 3inputs = tokenizer( 4 prompt, 5 return_tensors="pt" 6).to(model.device) 7 8outputs = model.generate( 9 **inputs, 10 max_new_tokens=120 11) 12 13print( 14 tokenizer.decode( 15 outputs[0], 16 skip_special_tokens=True 17 ) 18)
Step 3 — Serve with vLLM
1from vllm import LLM, SamplingParams 2 3llm = LLM( 4 model=model_name 5) 6 7params = SamplingParams( 8 temperature=0.8, 9 max_tokens=150 10) 11 12response = llm.generate( 13 ["What is a Transformer?"], 14 params 15) 16 17print(response[0].outputs[0].text)
What You'll Learn
- Load a quantized LLM.
- Reduce memory usage with 4-bit quantization.
- Perform efficient inference.
- Serve an optimized model with vLLM.
Best Practices
| Recommendation | Benefit |
|---|---|
| Choose INT8 or INT4 based on accuracy requirements | Balance quality and efficiency |
| Use GPTQ or AWQ for post-training quantization | Minimal quality loss |
| Use GGUF for local CPU deployments | Portable and efficient inference |
| Export to ONNX for hardware interoperability | Flexible deployment |
| Use TensorRT on NVIDIA GPUs | Maximum inference performance |
| Deploy production APIs with vLLM | High throughput and low latency |
| Use tensor or pipeline parallelism for very large models | Scale across multiple GPUs |
| Benchmark latency, throughput, and memory usage | Make data-driven deployment decisions |
Module Summary
After completing this module, you will be able to:
- Explain why optimization is essential for deploying modern Transformer models.
- Apply quantization techniques to reduce memory usage and improve inference speed.
- Compare GPTQ and AWQ for post-training quantization.
- Understand the GGUF format and its role in efficient local inference.
- Export Transformer models to ONNX for cross-platform deployment.
- Use TensorRT to optimize inference on NVIDIA GPUs.
- Serve LLMs efficiently with vLLM using continuous batching and optimized KV cache management.
- Understand tensor parallelism and pipeline parallelism for scaling large models across multiple GPUs.
- Explain speculative decoding and how it reduces generation latency.
- Deploy and benchmark a quantized LLM for production-ready inference.
Next Module: Module 26 – Advanced Transformer Architectures, where you'll explore Mixture of Experts (MoE), Switch Transformers, Sparse Attention, Longformer, BigBird, Performer, Mamba, RWKV, State Space Models (SSMs), and other modern architectures for scaling efficient sequence modeling.