Skip to workspace

RL Lab

Learn sequential decisions first. Then connect rewards and expert preferences to language-model post-training.

Learn the decision loop

Observation→ Policy→ Action→ Environment→ Reward + next observation ↺

Five states, two actions. Start at 0, reach terminal state 4. Every ordinary move earns −0.05; reaching the goal earns +1. Each episode is capped at 60 steps.

Train to inspect the action values and episode returns.

Choose the correct post-training method

MethodTraining signalWhat changes / what it needs
SFTReviewed target responseImitates targets; no preference pair or reward required
Reward modelingScores or comparison pairsFits a predictor of judgments; not the policy itself
RLHFHuman feedback, often learned rewardPolicy optimization; audit human labels and reward generalization
RLAIFAI-generated feedbackAudit judge calibration, provenance, and shared blind spots
DPOChosen / rejected pairsOffline preference optimization against a reference; no separate reward model required
Principle-based feedbackExplicit criteria for critique or rankingA feedback design, not a single optimizer
Rejection samplingGenerate then filter candidatesSelects outputs; weight updates are a separate step
Online RL / GRPONew rollouts + rewardNeeds generation budget, tested reward, and held-out evaluation

RLCD · research reading

This research track is being prepared. A relevant primary paper is RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment (ICLR 2024). This paper-specific method will be covered separately from the other alignment objectives.

Do not treat the acronym as another name for DPO, SFT, or an autoencoder.

Reward hacking exercise: if the only reward is valid JSON, a model can emit schema-valid nonsense. Define semantic checks and safety vetoes, then test adversarial counterexamples before optimizing.
Bring this lab to your teamOriginal teaching examples · No production training claims