Renaissance LabsLearn

Models · Safety and alignment

Running AI agents safely on a cluster

RL rollouts with internet access

13 minIntermediate12 chapters

Reinforcement learning agents learn by acting, often with a terminal and the internet. The film explains how a research cluster works, what an RL rollout looks like, eight concrete failure modes, and a safety design: strong sandboxes, default-deny networking through a proxy, logging outside the box, canary tokens, a kill switch and a staged rollout.

What to take away

  • On a cluster your job runs as you, and the machine is shared, so you are responsible for what your agents do.
  • A sandbox is only as safe as what is mounted in it, so a good rollout sandbox starts empty.
  • Never give one rollout private data, untrusted content and a way to send information out at the same time.
  • Training searches for whatever earns reward, so loopholes in the checker will be found and must be closed.
  • Move up from no network to a frozen web mirror to a small allowlist only after the lower stage has run clean.

Go deeper

Read the transcript63 lines, click any line to jump

Sources

  • Simon Willison on the lethal trifecta for AI agents
  • OWASP Top 10 for LLM Applications
  • MacDiarmid et al. 2025, Natural emergent misalignment from reward hacking in production RL (Anthropic)
  • Baker et al. 2025, monitoring reasoning models for misbehavior (OpenAI)
  • Documentation for Slurm, Apptainer, gVisor and Firecracker, and Linux kernel docs

Where a source has no link, it is cited from the film's research notes. Estimates and unverified claims are labelled in the film.

Keep watching