RL Lab
Learn sequential decisions first. Then connect rewards and expert preferences to language-model post-training.
Learn the decision loop
Five states, two actions. Start at 0, reach terminal state 4. Every ordinary move earns −0.05; reaching the goal earns +1. Each episode is capped at 60 steps.
Train to inspect the action values and episode returns.
Choose the correct post-training method
| Method | Training signal | What changes / what it needs |
|---|---|---|
| SFT | Reviewed target response | Imitates targets; no preference pair or reward required |
| Reward modeling | Scores or comparison pairs | Fits a predictor of judgments; not the policy itself |
| RLHF | Human feedback, often learned reward | Policy optimization; audit human labels and reward generalization |
| RLAIF | AI-generated feedback | Audit judge calibration, provenance, and shared blind spots |
| DPO | Chosen / rejected pairs | Offline preference optimization against a reference; no separate reward model required |
| Principle-based feedback | Explicit criteria for critique or ranking | A feedback design, not a single optimizer |
| Rejection sampling | Generate then filter candidates | Selects outputs; weight updates are a separate step |
| Online RL / GRPO | New rollouts + reward | Needs generation budget, tested reward, and held-out evaluation |
RLCD · research reading
This research track is being prepared. A relevant primary paper is RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment (ICLR 2024). This paper-specific method will be covered separately from the other alignment objectives.
Do not treat the acronym as another name for DPO, SFT, or an autoencoder.