Video by Hugging Face via YouTube
Training Agents, Session 3: from a fine-tuned baseline to reinforcement learning with GRPO.
In this live session, we’ll move past imitation and start training on outcomes. We’ll take the model from the earlier sessions and run Group Relative Policy Optimization: sample multiple completions per prompt, score them with a reward function, and use the group’s relative scores as the training signal — no value model, no reward model, no critic to train. Everything runs in TRL.
What we’ll cover:
– Why GRPO after SFT, and what RL adds that imitation can’t
– How GRPO works: group sampling, relative advantages, and why that replaces a critic
– Writing verifiable reward functions for agent tasks
– Running TRL GRPO on Hugging Face Jobs
– Reading the training curves: reward, KL, and completion length
– Common failure modes: reward hacking, collapse, and length gaming
Repo: https://github.com/burtenshaw/training-agents
This is part of the Training Agents series: using coding agents to design, run, monitor, and review post-training experiments, while training models to become better agents.