Module 24 — Alignment: Reinforcement Learning for Large Language Models

Introduction

Large Language Models (LLMs) become powerful during pretraining, but pretraining alone does not guarantee that a model will be helpful, accurate, safe, or aligned with human preferences.

To improve model behavior, modern LLMs go through an alignment stage after pretraining. Alignment techniques teach models to:

• Follow user instructions
• Produce helpful responses
• Avoid harmful outputs
• Reduce hallucinations
• Improve reasoning
• Better reflect human preferences

Early alignment pipelines relied heavily on Reinforcement Learning from Human Feedback (RLHF). Today, simpler and more efficient methods such as Direct Preference Optimization (DPO), ORPO, and GRPO are also widely used.

Applications include:

• Chat Assistants
• Coding Assistants
• AI Tutors
• Customer Support
• Enterprise AI
• Medical AI
• Legal AI

In this module, you'll learn:

• Supervised Fine-Tuning (SFT)
• Reward Models
• RLHF
• DPO
• PPO
• ORPO
• GRPO
• Constitutional AI
• AI Safety
• Preference Optimization

Modern LLM Training Pipeline

textCopy
1              Pretraining
2                    │
3                    ▼
4         Supervised Fine-Tuning
5                    │
6                    ▼
7            Preference Data
8                    │
9                    ▼
10          Alignment Method
11      (RLHF / DPO / ORPO / GRPO)
12                    │
13                    ▼
14            Aligned Language Model

Why Alignment is Needed

A pretrained model predicts the next token well, but it may:

• Ignore user intent
• Produce unsafe content
• Hallucinate facts
• Generate inconsistent responses
• Fail to follow instructions

Alignment teaches the model how people prefer it to respond, not just what text is statistically likely.

1. Supervised Fine-Tuning (SFT)

What is SFT?

Supervised Fine-Tuning is usually the first alignment stage.

The model is trained on high-quality instruction–response examples.

Dataset Example

textCopy
1Instruction:
2
3Explain recursion.
4
5↓
6
7Response:
8
9Recursion is a function calling itself...

Pipeline

textCopy
1Instruction Dataset
2
3↓
4
5Pretrained LLM
6
7↓
8
9Supervised Learning
10
11↓
12
13Instruction-Following Model

Advantages

• Simple
• Stable training
• Strong instruction following

Limitations

• Cannot directly optimize human preferences
• Limited by demonstration quality

Example Dataset

pythonCopy
1sample = {
2    "instruction": "Summarize this article.",
3    "input": "...",
4    "output": "Short summary..."
5}

2. Reward Models

What is a Reward Model?

A reward model learns to predict which response humans prefer.

Instead of generating text, it outputs a preference score.

Pipeline

textCopy
1Prompt
2
3↓
4
5Response A
6
7Response B
8
9↓
10
11Human Preference
12
13↓
14
15Reward Model

Example

textCopy
1Question:
2
3Explain gravity.
4
5Response A
6
7Helpful
8
9Reward = 0.92
10
11Response B
12
13Incorrect
14
15Reward = 0.18

Reward models provide the optimization signal used in RLHF.

3. Reinforcement Learning from Human Feedback (RLHF)

What is RLHF?

RLHF combines reinforcement learning with human preference data.

Typical workflow

textCopy
1Prompt
2
3↓
4
5LLM
6
7↓
8
9Generated Response
10
11↓
12
13Reward Model
14
15↓
16
17Policy Optimization
18
19↓
20
21Improved LLM

Training stages

1. Pretraining
2. Supervised Fine-Tuning
3. Reward Model Training
4. Reinforcement Learning

Advantages

• Optimizes for human preferences
• Improves helpfulness
• Better conversational behavior

Limitations

• Complex pipeline
• Requires reward model
• Computationally expensive

4. PPO (Proximal Policy Optimization)

What is PPO?

PPO is the reinforcement learning algorithm most famously associated with early RLHF pipelines.

Objective

• Improve responses
• Prevent overly large policy updates
• Maintain training stability

Pipeline

textCopy
1Current Policy
2
3↓
4
5Generate Responses
6
7↓
8
9Reward
10
11↓
12
13PPO Update
14
15↓
16
17New Policy

Advantages

• Stable optimization
• Well-studied RL algorithm

Limitations

• Expensive
• Requires reward model
• Many hyperparameters

5. Direct Preference Optimization (DPO)

What is DPO?

DPO directly learns from chosen vs. rejected response pairs.

Unlike RLHF, DPO does not require a separate reinforcement learning stage.

Dataset

textCopy
1Prompt
2
3↓
4
5Chosen Response
6
7↓
8
9Rejected Response

Pipeline

textCopy
1Preference Dataset
2
3↓
4
5DPO Objective
6
7↓
8
9Aligned Model

Advantages

• Simpler than RLHF
• Stable training
• No PPO optimization loop
• No online reward optimization

Example Preference Pair

pythonCopy
1sample = {
2    "prompt": "Explain attention.",
3    "chosen": "Attention allows...",
4    "rejected": "Attention is..."
5}

6. ORPO (Odds Ratio Preference Optimization)

What is ORPO?

ORPO combines supervised fine-tuning and preference optimization into a single training objective.

Pipeline

textCopy
1Instruction Data
2
3+
4
5Preference Data
6
7↓
8
9Unified Optimization
10
11↓
12
13Aligned Model

Advantages

• Single-stage optimization
• Simpler training
• Competitive performance
• Reduced computational cost

7. GRPO (Group Relative Preference Optimization)

What is GRPO?

GRPO optimizes preferences by comparing multiple candidate responses for the same prompt instead of relying only on a single chosen/rejected pair.

Concept

textCopy
1Prompt
2
3↓
4
5Response A
6
7Response B
8
9Response C
10
11↓
12
13Relative Ranking
14
15↓
16
17Optimization

Advantages

• Better utilization of preference data
• Useful for reasoning models
• Efficient preference learning

Comparison of Alignment Methods

8. Constitutional AI

What is Constitutional AI?

Constitutional AI aligns a model using a predefined set of principles or rules (a "constitution") rather than relying exclusively on human labels.

Example Constitution

textCopy
1- Be truthful.
2- Avoid harmful content.
3- Respect user privacy.
4- Explain uncertainty.
5- Avoid unsupported claims.

Workflow

textCopy
1Prompt
2
3↓
4
5Model Response
6
7↓
8
9Self-Critique
10
11↓
12
13Revision
14
15↓
16
17Improved Response

Advantages

• Scalable supervision
• Encourages consistent behavior
• Reduces reliance on manual annotation

9. AI Safety

What is AI Safety?

AI safety focuses on making models reliable, secure, and aligned with intended behavior.

Goals

• Reduce harmful outputs
• Prevent misuse
• Improve robustness
• Handle uncertainty responsibly
• Protect privacy
• Resist prompt injection and jailbreak attempts

Common Techniques

• Alignment training
• Safety evaluations
• Red teaming
• Guardrails
• Content moderation
• Grounded responses
• Human oversight

End-to-End Alignment Pipeline

textCopy
1Raw Text
2
3↓
4
5Pretraining
6
7↓
8
9SFT
10
11↓
12
13Preference Dataset
14
15↓
16
17DPO / RLHF / ORPO / GRPO
18
19↓
20
21Evaluation
22
23↓
24
25Deployment

Practice — Preference Optimization

Step 1 — Load a Preference Dataset

pythonCopy
1from datasets import Dataset
2
3dataset = Dataset.from_dict({
4    "prompt": [
5        "Explain machine learning."
6    ],
7    "chosen": [
8        "Machine learning is a field of AI that enables systems to learn from data."
9    ],
10    "rejected": [
11        "Machine learning is only about robots."
12    ]
13})
14
15print(dataset)

Step 2 — Configure a DPO Trainer

pythonCopy
1from transformers import (
2    AutoModelForCausalLM,
3    AutoTokenizer
4)
5from trl import DPOTrainer, DPOConfig
6
7model_name = "meta-llama/Llama-3.2-1B"
8
9tokenizer = AutoTokenizer.from_pretrained(model_name)
10
11model = AutoModelForCausalLM.from_pretrained(model_name)
12
13config = DPOConfig(
14    output_dir="./dpo-model",
15    per_device_train_batch_size=2,
16    learning_rate=5e-6,
17    num_train_epochs=1
18)

Step 3 — Train with DPO

pythonCopy
1trainer = DPOTrainer(
2    model=model,
3    args=config,
4    train_dataset=dataset,
5    processing_class=tokenizer
6)
7
8trainer.train()
9
10trainer.save_model("./aligned-model")

What You'll Learn

• Prepare preference datasets.
• Understand chosen vs. rejected responses.
• Configure a DPO training pipeline.
• Align an instruction-following model using preference optimization.

Choosing the Right Alignment Method

Best Practices

Module Summary

After completing this module, you will be able to:

• Explain why alignment is a critical stage after LLM pretraining.
• Understand the role of Supervised Fine-Tuning (SFT) in instruction following.
• Describe how reward models capture human preferences.
• Explain the RLHF pipeline and the role of PPO in policy optimization.
• Compare DPO, ORPO, and GRPO as modern preference optimization techniques.
• Understand the principles behind Constitutional AI and AI safety.
• Prepare preference datasets and perform Direct Preference Optimization (DPO).
• Select an appropriate alignment strategy based on project goals, available data, and computational resources.

Next Module: Module 25 – LLM Inference & Deployment, where you'll learn quantization (INT8, INT4, GGUF), vLLM, Text Generation Inference (TGI), TensorRT-LLM, ONNX Runtime, FastAPI serving, streaming generation, and production deployment of large language models.