Converting GGUF Models to Ollama: A Complete Beginner's Guide

Introduction

Large Language Models can now be executed entirely on local hardware using GGUF models and Ollama. GGUF is a highly optimized model format used by llama.cpp, while Ollama provides a simple API and chat interface for deploying these models.

In this tutorial, we will learn how to:

• Import a GGUF model into Ollama
• Create a Modelfile
• Build a custom Ollama model
• Run the model locally
• Access the model through the Ollama API
• Use the model in RAG applications

---

What is GGUF?

GGUF is a compressed model format designed for efficient inference.

Features:

• Fast CPU inference
• GPU acceleration support
• Quantized models (Q4, Q5, Q8)
• Lower memory requirements
• Compatible with llama.cpp and Ollama

Popular GGUF models:

• Qwen 2/2.5/3
• Llama 3
• Gemma
• Mistral
• DeepSeek

---

What is Ollama?

Ollama is a local LLM runtime that provides:

• Chat interface
• REST API
• Model management
• GPU support
• Integration with LangChain and RAG systems

Ollama internally uses GGUF models.

---

Step 1: Prepare the Model Directory

Create a directory:

bashCopy
1mkdir mymodel
2cd mymodel

Copy your GGUF model:

bashCopy
1cp /path/to/model.gguf .

Example:

bashCopy
1Qwen3-1.7B-Q4_K_M.gguf

---

Step 2: Create the Modelfile

Create a new file:

bashCopy
1nano Modelfile

Example:

textCopy
1FROM ./model.gguf
2
3TEMPLATE """{{ .Prompt }}"""
4
5PARAMETER temperature 0.7
6PARAMETER num_ctx 4096
7
8SYSTEM """
9You are a helpful AI assistant.
10"""

---

Step 3: Build the Ollama Model

Run:

bashCopy
1ollama create qwen3-local -f Modelfile

Expected output:

textCopy
1gathering model components
2parsing GGUF
3writing manifest
4success

---

Step 4: Run the Model

Interactive mode:

bashCopy
1ollama run qwen3-local

Single prompt:

bashCopy
1ollama run qwen3-local "Explain transformers."

---

Step 5: View Installed Models

bashCopy
1ollama list

Example:

textCopy
1NAME            SIZE
2qwen3-local     1.4 GB

Remove a model:

bashCopy
1ollama rm qwen3-local

---

Using the Ollama API

Start the server:

bashCopy
1ollama serve

Default API:

textCopy
1http://localhost:11434

Test:

bashCopy
1curl http://localhost:11434/api/generate \
2-d '{
3    "model":"qwen3-local",
4    "prompt":"Hello",
5    "stream":false
6}'

Response:

jsonCopy
1{
2    "response": "Hello! How can I help you?"
3}

---

Accessing Ollama from Another Device

Allow external access:

bashCopy
1export OLLAMA_HOST=0.0.0.0:11434
2ollama serve

Find your local IP:

bashCopy
1ip addr

Example:

textCopy
1192.168.1.100

Remote applications can now access:

textCopy
1http://192.168.1.100:11434/api/generate

---

Flask Example

pythonCopy
1import requests
2
3response = requests.post(
4    "http://192.168.1.100:11434/api/generate",
5    json={
6        "model": "qwen3-local",
7        "prompt": "Hello",
8        "stream": False
9    }
10)
11
12print(response.json())

---

GPU or CPU?

Check GPU usage:

bashCopy
1nvidia-smi

Check Ollama:

bashCopy
1ollama ps

Example:

textCopy
1NAME          PROCESSOR
2qwen3-local   100% GPU

---

GGUF vs Ollama

---

Typical RAG Architecture

textCopy
1Hugging Face Model
2        ↓
3Convert to GGUF
4        ↓
5Import into Ollama
6        ↓
7Ollama API
8        ↓
9LangChain
10        ↓
11Vector Database
12        ↓
13Flask / FastAPI / Streamlit
14        ↓
15RAG Application

---

Conclusion

GGUF and Ollama provide a powerful solution for running large language models locally. Developers can build AI assistants, RAG systems, chatbots, document analyzers, and private AI applications without relying on cloud APIs.

With just a GGUF model and Ollama, modern LLM applications can run efficiently on consumer hardware, including laptops with NVIDIA GPUs.