Module 26 — Model Serving

Introduction

Training a Transformer model is only half of the machine learning lifecycle. To make the model useful for real-world applications, it must be deployed as a scalable, reliable, and secure inference service.

Model Serving is the process of exposing trained models through APIs or inference servers so applications can send requests and receive predictions in real time.

Modern Transformer deployment solutions include:

• Hugging Face Inference
• FastAPI
• TorchServe
• NVIDIA Triton Inference Server
• vLLM
• Ollama
• Docker
• Kubernetes

These technologies are used to deploy:

• AI Chatbots
• Code Assistants
• Search Systems
• RAG Applications
• Vision Models
• Speech Recognition Systems
• Enterprise AI APIs

In this module, you'll learn:

• Hugging Face Inference
• FastAPI
• TorchServe
• Triton Inference Server
• vLLM
• Ollama
• Docker
• Kubernetes
• API Deployment
• Monitoring
• Deploy a Chat API

Complete Model Serving Pipeline

textCopy
1              Trained Model
2                    │
3                    ▼
4             Inference Engine
5     (vLLM / TorchServe / Triton)
6                    │
7                    ▼
8              FastAPI Server
9                    │
10                    ▼
11          Docker Container
12                    │
13                    ▼
14            Kubernetes Cluster
15                    │
16                    ▼
17              Client Applications

1. Hugging Face Inference

What is Hugging Face Inference?

Hugging Face provides hosted inference services that allow you to use pretrained models through a simple API without managing infrastructure.

Features

• Hosted inference
• REST API
• Large model support
• Automatic scaling (depending on the service)
• Easy integration

Architecture

textCopy
1Client
2
3↓
4
5Hugging Face API
6
7↓
8
9Hosted Model
10
11↓
12
13Prediction

Example

pythonCopy
1from huggingface_hub import InferenceClient
2
3client = InferenceClient(
4    model="meta-llama/Llama-3.2-1B"
5)
6
7response = client.text_generation(
8    "Explain Transformers.",
9    max_new_tokens=100
10)
11
12print(response)

2. FastAPI

What is FastAPI?

FastAPI is a modern Python framework used to build high-performance REST APIs for machine learning inference.

Advantages

• Fast
• Async support
• Automatic OpenAPI documentation
• Easy deployment
• Type validation

Architecture

textCopy
1Client
2
3↓
4
5FastAPI
6
7↓
8
9Transformer Model
10
11↓
12
13Prediction

Example

pythonCopy
1from fastapi import FastAPI
2from transformers import pipeline
3
4app = FastAPI()
5
6generator = pipeline(
7    "text-generation",
8    model="gpt2"
9)
10
11@app.post("/generate")
12def generate(prompt: str):
13    output = generator(
14        prompt,
15        max_new_tokens=100
16    )
17
18    return {
19        "response": output[0]["generated_text"]
20    }

3. TorchServe

What is TorchServe?

TorchServe is an official PyTorch model serving framework for deploying PyTorch models at scale.

Features

• Model versioning
• Multi-model serving
• Batch inference
• Metrics collection
• REST APIs

Pipeline

textCopy
1PyTorch Model
2
3↓
4
5TorchServe
6
7↓
8
9REST API

Advantages

• Production ready
• Optimized for PyTorch
• Monitoring support

4. Triton Inference Server

What is Triton?

NVIDIA Triton Inference Server is a high-performance serving platform that supports multiple deep learning frameworks.

Supported Frameworks

• PyTorch
• TensorFlow
• ONNX Runtime
• TensorRT
• Python Backend

Architecture

textCopy
1Client
2
3↓
4
5Triton
6
7↓
8
9GPU
10
11↓
12
13Prediction

Advantages

• Dynamic batching
• Multi-model serving
• GPU optimization
• High throughput

5. vLLM

What is vLLM?

vLLM is a high-throughput inference engine built specifically for Large Language Models.

Key Features

• Continuous batching
• PagedAttention
• Efficient KV cache
• Streaming generation
• OpenAI-compatible API

Architecture

textCopy
1Requests
2
3↓
4
5Continuous Batch
6
7↓
8
9PagedAttention
10
11↓
12
13LLM
14
15↓
16
17Responses

Advantages

• High throughput
• Low latency
• Efficient GPU memory utilization

Example

pythonCopy
1from vllm import LLM, SamplingParams
2
3llm = LLM(
4    model="meta-llama/Llama-3.2-1B"
5)
6
7params = SamplingParams(
8    temperature=0.7,
9    max_tokens=128
10)
11
12result = llm.generate(
13    ["Explain self-attention."],
14    params
15)
16
17print(result[0].outputs[0].text)

6. Ollama

What is Ollama?

Ollama is a lightweight runtime for running open-source language models locally.

Features

• Local inference
• GGUF support
• Simple CLI
• REST API
• Easy model management

Pipeline

textCopy
1GGUF Model
2
3↓
4
5Ollama
6
7↓
8
9REST API

Advantages

• Beginner friendly
• Offline inference
• Cross-platform

Example

bashCopy
1ollama run llama3.2

Python API

pythonCopy
1import requests
2
3response = requests.post(
4    "http://localhost:11434/api/generate",
5    json={
6        "model": "llama3.2",
7        "prompt": "Explain transformers."
8    }
9)
10
11print(response.json())

Comparison of Inference Engines

7. Docker

Why Docker?

Docker packages the application and all its dependencies into a portable container.

Architecture

textCopy
1Application
2
3↓
4
5Docker Image
6
7↓
8
9Docker Container

Advantages

• Reproducible environments
• Easy deployment
• Platform independent
• Simplified dependency management

Dockerfile

dockerfileCopy
1FROM python:3.11-slim
2
3WORKDIR /app
4
5COPY . .
6
7RUN pip install -r requirements.txt
8
9CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Build

bashCopy
1docker build -t chat-api .

Run

bashCopy
1docker run -p 8000:8000 chat-api

8. Kubernetes

What is Kubernetes?

Kubernetes is a container orchestration platform used to deploy and manage applications across clusters.

Responsibilities

• Scheduling
• Scaling
• Load balancing
• Self-healing
• Rolling updates

Architecture

textCopy
1Docker Container
2
3↓
4
5Kubernetes Pod
6
7↓
8
9Service
10
11↓
12
13Load Balancer

Advantages

• High availability
• Automatic scaling
• Fault tolerance

Example Deployment

yamlCopy
1apiVersion: apps/v1
2kind: Deployment
3
4metadata:
5  name: chat-api
6
7spec:
8  replicas: 2
9
10  selector:
11    matchLabels:
12      app: chat-api
13
14  template:
15    metadata:
16      labels:
17        app: chat-api
18
19    spec:
20      containers:
21      - name: chat-api
22        image: chat-api:latest

9. API Deployment

Deployment Workflow

textCopy
1Train Model
2
3↓
4
5Save Model
6
7↓
8
9FastAPI
10
11↓
12
13Docker
14
15↓
16
17Kubernetes
18
19↓
20
21Public REST API

Best Practices

• Version your models
• Secure endpoints with authentication
• Use HTTPS
• Enable request logging
• Implement rate limiting
• Validate inputs
• Handle errors gracefully

10. Monitoring

Why Monitoring Matters

Monitoring helps ensure that deployed models remain healthy, responsive, and accurate over time.

Common Metrics

Monitoring Workflow

textCopy
1API
2
3↓
4
5Metrics
6
7↓
8
9Dashboard
10
11↓
12
13Alerts

Popular Monitoring Tools

• Prometheus
• Grafana
• OpenTelemetry
• Loki
• ELK Stack

Practice — Deploy a Chat API

Step 1 — Create a FastAPI Server

pythonCopy
1from fastapi import FastAPI
2from transformers import pipeline
3
4app = FastAPI()
5
6chatbot = pipeline(
7    "text-generation",
8    model="gpt2"
9)
10
11@app.post("/chat")
12def chat(prompt: str):
13
14    response = chatbot(
15        prompt,
16        max_new_tokens=100
17    )
18
19    return {
20        "reply": response[0]["generated_text"]
21    }

Step 2 — Run the API

bashCopy
1uvicorn main:app --reload

Test

textCopy
1POST /chat
2
3{
4    "prompt": "Explain Transformers."
5}

Response

jsonCopy
1{
2    "reply": "Transformers are deep learning models..."
3}

Step 3 — Dockerize the API

dockerfileCopy
1FROM python:3.11
2
3WORKDIR /app
4
5COPY . .
6
7RUN pip install -r requirements.txt
8
9CMD [
10    "uvicorn",
11    "main:app",
12    "--host",
13    "0.0.0.0",
14    "--port",
15    "8000"
16]

Build

bashCopy
1docker build -t transformer-api .

Run

bashCopy
1docker run -p 8000:8000 transformer-api

Step 4 — Deploy with Kubernetes

bashCopy
1kubectl apply -f deployment.yaml

Verify

bashCopy
1kubectl get pods

What You'll Learn

• Build a REST API using FastAPI.
• Serve Transformer models through HTTP endpoints.
• Containerize the application with Docker.
• Deploy the API using Kubernetes.
• Monitor the deployment for performance and reliability.

Best Practices

Module Summary

After completing this module, you will be able to:

• Explain the complete lifecycle of serving Transformer models in production.
• Compare Hugging Face Inference, FastAPI, TorchServe, Triton Inference Server, vLLM, and Ollama.
• Build REST APIs for Transformer inference using FastAPI.
• Containerize inference services with Docker.
• Deploy scalable services using Kubernetes.
• Expose models through secure production APIs.
• Monitor latency, throughput, and infrastructure health.
• Deploy an end-to-end chat API suitable for real-world applications.

Next Module: Module 27 – Advanced Transformer Research, where you'll explore Mixture of Experts (MoE), Sparse Transformers, Long-Context Models, State Space Models (SSMs), Mamba, RWKV, Retrieval-Augmented Transformers, and emerging research directions shaping the next generation of AI systems.