Awesome Post-Training

A math-first, systems-aware guide to post-training foundation models, with reinforcement learning as the main thread.

Post-training is where a pretrained model is turned into a useful policy: it learns instructions, preferences, reasoning strategies, tool use, and safety boundaries. This repository connects the full stack:

flowchart LR
  P[Prompts / environments] --> R[Actor rollouts]
  R --> F[Humans, reward models, verifiers, tools]
  F --> A[Returns / advantages / preferences]
  A --> U[SFT, DPO, PPO, RLOO, GRPO]
  U --> E[Capability + safety evaluation]
  E --> D[Deployment]
  D --> P

It is a learning map, not a leaderboard and not an endorsement list. Entries distinguish papers from implementations, open source from hosted products, and established results from fast-moving research.

Start here

Goal	Recommended path
Build the mathematical foundation	Math primer → Sutton & Barto → REINFORCE → GAE → PPO
Understand modern LLM RL	Algorithms → DeepSeekMath/GRPO → DeepSeek-R1 → DAPO
Implement a training system	Systems → framework comparison
Choose data and evaluations	Datasets & evaluation
Prepare for interviews	Interview guide → learning roadmap
Track the field	Papers, companies & products, field signals

The post-training map

Stage	Input	Learning signal	Typical method	Main failure mode
Continued pretraining	Unlabeled tokens	Next-token targets	Cross-entropy	Domain gain but behavior drift
Supervised fine-tuning	Prompt-response demonstrations	Reference tokens	SFT / instruction tuning	Imitates mistakes; exposure bias
Preference modeling	Chosen/rejected responses	Human or AI comparisons	Bradley–Terry reward model	Annotator and proxy bias
Offline alignment	Static preference pairs	Relative preference	DPO, IPO, KTO, ORPO, SimPO	Distribution mismatch
Online policy optimization	Policy rollouts	RM/human/environment reward	PPO, RLOO	Reward hacking; high systems cost
Verifiable RL	Multiple sampled solutions	Tests/exact answers/verifiers	GRPO, DAPO-style training	Sparse reward; shortcut exploitation
Process supervision	Reasoning steps or trajectories	Step-level reward	PRM, search, credit assignment	Incorrect intermediate labels
Distillation	Teacher samples/logits	Token or sequence targets	Offline/on-policy distillation	Teacher ceiling; mode collapse
Safety alignment	Adversarial prompts/policies	Preferences/rules/constitutions	RLAIF, red-teaming, RLHF	Over-refusal or jailbreaks

The four objectives interviewers expect you to separate

SFT maximizes demonstration likelihood:

$$\mathcal L_{\mathrm{SFT}}(\theta)=-\mathbb E_{(x,y)\sim D}\sum_t\log\pi_\theta(y_t\mid x,y_{<t}).$$
Reward modeling fits preferences, commonly with a Bradley–Terry likelihood:

$$P(y_w\succ y_l\mid x)=\sigma\!\left(r_\phi(x,y_w)-r_\phi(x,y_l)\right).$$
KL-regularized RL improves expected reward without moving arbitrarily far from a reference policy:

$$\max_\pi\;\mathbb E_{x,y\sim\pi}[r(x,y)]-\beta\,\mathbb E_x D_{\mathrm{KL}}(\pi(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)).$$
Direct preference optimization optimizes a classification objective implied by the regularized optimum, without an explicit online RL loop. See the derivation in the math primer.

Algorithm chooser

Situation	Good first baseline	Why	Watch closely
High-quality demonstrations, little preference data	SFT	Simple and stable	Validation loss is not task quality
Static chosen/rejected pairs	DPO	Few moving parts	Pair quality and reference-policy mismatch
Scalar feedback, online rollouts, learned RM	PPO or RLOO	General sequence reward	KL, reward overoptimization, critic error
Verifiable answer and several samples per prompt	GRPO-family	Group baseline avoids a critic	Zero-variance groups, length bias, clipping
Expensive teacher, cheap student	On-policy distillation	Student visits its own states	Teacher sampling cost and tokenizer mismatch
Multi-turn tools or code execution	Agentic RL	Real environment feedback	Non-stationarity, timeouts, sandboxing, credit assignment

There is no universally best optimizer. The reward, sampling distribution, group construction, token aggregation, and systems implementation often matter as much as the algorithm label.

Frameworks at a glance

Framework	Best fit	Training / rollout character
TRL	Accessible single-node and research workflows	Transformers-native trainers; broad algorithm coverage
verl	Flexible large-scale RL research	Ray orchestration; multiple training and rollout backends
OpenRLHF	Distributed RLHF recipes	Ray, vLLM, DeepSpeed/Megatron options
NeMo RL	NVIDIA-scale multimodal post-training	DTensor/Megatron with vLLM, SGLang, or Megatron rollout
slime	RL scaling and custom agentic generation	Megatron/FSDP plus SGLang; data-buffer architecture
SkyRL	Agentic and long-horizon RL research	Environments, generators, distributed training
ART	Application-level agent RL	Lightweight integration around GRPO-style training
Tinker	Hosted low-level post-training primitives	Remote sampling and forward/backward API

See frameworks.md before selecting one; APIs and support matrices change quickly.

Important paper trails

RL foundations: REINFORCE → TRPO → GAE → PPO.
Human feedback: Deep RL from Human Preferences → Learning to Summarize → InstructGPT → Constitutional AI.
Offline preferences: DPO → IPO/KTO/ORPO/SimPO and online preference variants.
Reasoning RL: DeepSeekMath (GRPO) → DeepSeek-R1 → DAPO and subsequent stabilization/system work.
Reward quality: reward overoptimization → process supervision/PRM800K → RewardBench → judge reliability.

The curated reading order and the distinction between foundational and frontier work are in papers.md.

Repository structure

.
├── README.md
├── catalog/
│   ├── resources.yaml       # machine-readable source of truth
│   └── schema.json
├── docs/
│   ├── math-primer.md
│   ├── algorithms.md
│   ├── systems.md
│   ├── frameworks.md
│   ├── papers.md
│   ├── datasets-evaluation.md
│   ├── companies-products.md
│   ├── safety.md
│   ├── interview-guide.md
│   ├── learning-roadmap.md
│   └── catalog-report.md     # generated; do not hand-edit
└── scripts/                  # validation, report, link maintenance

Maintain the catalog

make setup
make report
make validate
make external-links          # networked, best effort

Each catalog record has a stable ID, type, stages, access mode, maturity, review date, and a concise reason for inclusion. The generated catalog report makes coverage gaps visible.

Scope and curation policy

Included resources must help explain or implement post-training. A paper should be influential, clarifying, reproducible, or expose a meaningful failure mode. A product entry must link to first-party material and state what is actually available. “SOTA” claims are time-scoped and never treated as permanent facts.

This repository does not reproduce paper abstracts, recommend training on unlicensed data, or treat reward scores as proof of alignment. See CONTRIBUTING.md and safety.md.

License

The curated list and original documentation are released under CC0-1.0. Linked resources retain their own licenses and terms.