Video by Hugging Face via YouTube
Training Agents, Session 4: from reward functions to environments.
In this live session, the reward stops being a function and becomes a place the agent acts in. In Session 3 the model answered once and a Python function scored it. Today the model runs a loop: open a file, run a command, read the error, try again. The training signal comes from what happened after acting.
We’ll walk the reset()/step() contract from Gym to LLM agents, build an environment with OpenEnv, push it to the Hub, and plug it into TRL’s GRPOTrainer with environment_factory. Then we go one step further and train a real coding agent harness through Harbor via OpenEnv, with AsyncGRPOTrainer on Hugging Face sandboxes.
What we’ll cover:
– Why verifiers stop working once an agent takes many actions, and why the reward has to live where the actions happen
– The reset()/step() contract: from CartPole to a coding agent with a shell, files, and tests
Anatomy of an RL environment: task, tools, backend, reward, episode
– The ecosystem: OpenEnv, Verifiers, OpenReward, Harbor, and how OpenEnv imports the others
– Building and serving an env with the openenv CLI: init, push, pull, fork
– Training with TRL: environment_factory and get_reward, no reward_funcs
– Demo 1: Qwen3-1.7B on MBPP inside a live Python session, held-out pass rate 0.49 → 0.59
– Demo 2: a real coding agent (OpenCode) on DeepCoder problems in HF sandboxes, reward 0.27 → 0.71 in 10 steps
– Reward hacking moves into the env: the try/except that never failed, the CVE fix read from .git history, and how to lock the box down