ludwig-ai--ludwig/examples/alignment

LLM Alignment with DPO and KTO

This example shows how to align a large language model with human preferences using Ludwig's built-in preference learning trainers. Alignment training is typically applied after an initial supervised fine-tuning (SFT) stage to improve response quality, reduce harmful outputs, and teach the model to follow instructions more reliably.

What is alignment?

Alignment refers to the process of shaping a model's behaviour to match human values and preferences. The classic approach — Reinforcement Learning from Human Feedback (RLHF) — requires training a separate reward model on human-ranked responses, then running a full RL loop (PPO) against it. Modern preference learning methods like DPO bypass the reward model entirely, making alignment cheaper and more stable to train.

When to use each trainer

Trainer	Data format	Use case	Compute
dpo	prompt, chosen, rejected	Preference pairs from human feedback; most widely studied	Medium — requires forward passes through both policy and reference model
kto	prompt, response, label (bool)	Single-label feedback (thumbs up/down); no paired responses needed	Low — simpler loss than DPO
orpo	prompt, chosen, rejected	Single-stage SFT + alignment; no separate reference model	Low — no reference model forward passes
grpo	prompt, custom reward function	RL-style training with a group-normalised reward signal; used in DeepSeek-R1	High — requires multiple rollouts per prompt

Choose DPO when you have human-ranked response pairs and want the best-studied approach. Choose KTO when collecting binary per-response feedback is easier than pairwise comparisons. Choose ORPO when you want to skip the SFT stage and align in one shot. Choose GRPO when you have a programmatic reward function (e.g. code execution, math verification).

Prerequisites

• GPU with at least 40 GiB of VRAM (A100 recommended)
• HuggingFace API Token
• Access approval to Llama-3.1-8B

Quick start

Install dependencies:

pip install "ludwig[llm]" datasets

Set your HuggingFace token:

export HUGGING_FACE_HUB_TOKEN="<your_token>"

Prepare the dataset:

python prepare_dataset.py

Run DPO training:

python train_dpo.py
# or with the CLI:
ludwig train --config config_dpo.yaml --dataset train.csv

Run KTO training:

ludwig train --config config_kto.yaml --dataset train_kto.csv

Run GRPO training (reuses the DPO preference-pair format):

python train_grpo.py
# or with the CLI:
ludwig train --config config_grpo.yaml --dataset preference_data.parquet

GRPO specifics

GRPO (Group Relative Policy Optimization, Shao et al. 2024) is the alignment method used by DeepSeek-R1. For each prompt it samples a group of grpo_num_generations completions, scores them, normalises rewards within the group, and applies a PPO-style clipped objective — without a separate critic model.

Ludwig's GRPO trainer consumes the same prompt / chosen / rejected columns as DPO, so a programmatic reward function is implemented as a pre-processing step: score each candidate completion in your dataset preparation pipeline, then emit the top-scoring completion as chosen and the lowest as rejected. See config_grpo.yaml for the full list of knobs (grpo_beta for the KL penalty, grpo_epsilon for PPO clipping, grpo_num_generations for the group size).

Files

File	Description
prepare_dataset.py	Downloads Anthropic/hh-rlhf and converts it to Ludwig format
train_dpo.py	DPO training script using the Python API
train_grpo.py	GRPO training script using the Python API
config_dpo.yaml	Ludwig config for DPO
config_kto.yaml	Ludwig config for KTO
config_orpo.yaml	Ludwig config for ORPO
config_grpo.yaml	Ludwig config for GRPO
alignment_dpo.ipynb	Colab-compatible notebook covering DPO, KTO evaluation, and upload

Upload to HuggingFace

After training, upload the aligned model:

ludwig upload hf_hub -r <your_org>/<model_name> -m results/experiment_run/model