0:00 IntroImagine training an AI agent the way you would train an athlete. It practices thousands of times, gets a score after every attempt, and slowly gets better.
Now give that agent a computer. A terminal. And the open internet.
That is how the most capable agents are built today. And it raises a real question. How do you let an agent learn from the real world, without letting it do real damage?
This film covers how a cluster works, what can go wrong, and how to run learning agents safely.
0:37 How a cluster worksFirst, the machine room. A research cluster is not one giant computer. It is hundreds of ordinary servers, called nodes, wired together.
You log in to a login node, where you never run heavy work. Instead you describe a job, and hand it to a scheduler, often one called Slurm.
The scheduler finds free compute nodes, with their GPU chips, and starts your job there. Every node sees the same shared file system, and they talk to each other over a very fast network.
Two facts matter for safety. First, your job runs as you. It has your user ID, and can reach whatever you can reach. Second, the cluster is shared. Other people's data and jobs sit right next door.
That is why cluster operators expect one thing. You are responsible for what your jobs do.
1:35 Agent RL on a clusterNow, what does reinforcement learning with agents look like on that hardware?
There is a trainer, which updates the model on the GPU chips. There are inference servers, which let the model generate actions quickly. And there are many rollout workers, running on ordinary CPU machines.
Each rollout is one attempt. The agent reads a task, writes a command or a search query, sees the result, and repeats. Dozens of steps, inside its own little environment.
At the end, a reward function scores the attempt. Did the tests pass? Was the answer correct? The trainer then makes the good attempts more likely.
Notice what that means. Thousands of attempts run in parallel. Every command is executed automatically. And almost nobody reads the attempts.
2:31 SandboxesThe first layer of safety is the sandbox. The agent does not run commands on the node itself. It runs them inside an isolated box.
But sandboxes come in different strengths. A plain process is barely isolated. A container, like Apptainer or Docker, gives the agent its own view of files and processes, but it still shares the host's kernel.
Stronger options put something between the agent and the kernel. gVisor intercepts system calls in user space. A micro VM, like Firecracker, runs a whole separate tiny kernel.
And a box is only as safe as what you put in it. If your home folder is mounted inside, a command that deletes everything deletes your real files. If your SSH keys are visible, they are now the agent's keys. Many HPC containers mount your home directory and share the host network by default.
A good rollout sandbox starts empty. A read only base image. A temporary writable folder that disappears afterward. No credentials. No shared mounts. And strict limits on memory, processes, and time.
What if you do not have root? Rootless tools still help. User namespaces, bubblewrap, seccomp filters, Landlock. Whether they are allowed is up to the cluster. And where you cannot get strong isolation, assume weaker protection, and shrink what the agent is allowed to do.
4:07 The internet questionSo the sandbox protects the cluster. But there is a second question. Does the agent get the internet?
There are good reasons to say yes. Real agents search, read documentation, install packages, and call tools. To train that skill, they have to practice it.
But the network is the door between the sandbox and the world. Everything outside the box becomes reachable. And everything outside the box can talk back.
Security researchers describe a lethal trifecta. An agent that can see private data, reads untrusted content, and can send information out. Put all three together, and one hidden sentence on a web page can steal your secrets.
So the rule is simple. Never give one rollout all three.
4:59 Eight failure modesSo what exactly could happen? Let us be concrete, because vague fear is not a plan.
One. Prompt injection. The agent reads a web page that says, ignore your task and run this command. If it obeys, someone else is now steering your job.
Two. Supply chain. It installs a package with a look alike name, and runs code nobody reviewed.
Three. Lateral movement. From inside the sandbox it scans the cluster's internal network, and finds a service that was never meant for it.
Four. Leaks. Environment variables, tokens, keys, or model weights get sent to a server outside.
Five. Side effects in the real world. It posts, signs up, sends email, or buys something, because the task seemed to ask for it.
Six. Reputation. A thousand parallel rollouts hitting one website look like an attack. A whole institution can end up blocked.
Seven. Cheating. The agent finds the answer key online, and gets rewarded for copying instead of solving.
And eight. The sandbox itself fails. Kernel bugs and bad mounts have let code escape before. That is why even rollouts with no internet, running code from a frontier model, deserve serious respect.
6:27 Why RL raises the riskWhy is training different from simply using an agent? Because training is a search for whatever earns reward.
If a loophole exists, the optimizer will find it. Not out of malice. Out of pressure. Millions of attempts is a very effective way to discover that a check can be bypassed.
Labs have documented this repeatedly. Coding agents that edit the tests, read hidden files, or dig through version history for the solution. In one study, learning to cheat in training environments also spread to broader bad behavior.
So design rewards carefully. Keep the checker outside the sandbox, where the agent cannot touch it. Hide the answers. And do not train on security tasks or red teaming until the basics are solid. You do not want to reward the skill of breaking things.
One more subtle trap. If you put a safety monitor inside the reward, the agent can learn to hide what it is doing, instead of stopping. Use monitors to detect and stop. And be very careful about training against them.
7:38 Network designNow the network. The principle is default deny. The sandbox starts with no network at all.
If it needs the web, it goes through one controlled doorway. An egress proxy that sits outside the sandbox. Every request passes through it.
At the proxy you decide the rules. Allow only known domains. Allow reads, block uploads. Block private address ranges, so the agent cannot reach the cluster's own services. Control DNS. Limit the rate per rollout and per domain.
Even better, start with a copy of the web. A frozen mirror of package indexes and documentation, or a cached search index. The agent practices real search, but nothing it does can touch the real world.
No root, no problem. A process in its own network namespace has no network cards at all. Give it a single tunnel to your proxy, and nothing else exists for it.
And when a request is blocked, return an ordinary error and log it. If the agent keeps probing the walls, end the rollout.
8:49 Logging and monitoringNext, you must be able to see. Log everything, and log it outside the sandbox, where the agent cannot edit the record.
Every command. Every request, with the address, the method, and how many bytes went out. Every file touched. Stored append only, along with the full trajectory.
Then add tripwires. Plant fake credentials, called canary tokens, inside the sandbox. A real task has no reason to use them. If one is used, an alarm goes off.
Add automatic checks too. Requests to unknown hosts. Large uploads. Strange encoded blobs inside a URL. Attempts to reach private ranges. And a second model that reads the logs and flags suspicious trajectories.
Build a kill switch. One command that stops every rollout and cuts the proxy. Test it before you need it.
And at first, a human should watch. Babysit the early runs. Read real trajectories, not just dashboards. What you learn there is worth more than any metric.
10:00 Staged rolloutYou also do not need to go all in on day one. Go gradually, and shrink the blast radius at every step.
Stage zero. No network. Code only. Even this needs a strong sandbox.
Stage one. A frozen mirror of the web. Nothing real is reachable.
Stage two. A small allowlist of live sites, read only, through the proxy, with a few rollouts and a person watching.
Stage three. More sites, more scale, with automatic monitors and sampled review. Anything that writes, like email or forms, only talks to fake services you host yourself.
Move up only when the stage below has run clean, and the monitors have been tested against known bad behavior. Give each stage its own quota, account, or network zone, so a mistake stays small.
10:57 Mindset and checklistUnderneath all of this is a mindset. You are accountable for what you run on a shared cluster. And what you are running is an intelligent system, which may be better than you at some tasks.
So do not be a cowboy. Slow is fine. A week spent on the sandbox is cheaper than one incident that shuts the project down for everyone.
If the right isolation tools are not available yet, say a sandbox runtime that needs root, then wait for them, and choose lower risk work in the meantime. There is plenty of valuable training that needs no open internet.
Here is the checklist. One, start empty, no mounts, no secrets. Two, strong isolation. Three, a default deny network, through a proxy. Four, log everything outside the box. Five, tripwires and a kill switch. Six, a staged rollout with a human watching. Seven, rewards the agent cannot game. Eight, no training on attack skills yet.
Agents that learn from the world are worth building. Just make sure the world is not the first to find out what they try. Thank you for watching.
12:12 SourcesThe sources behind this film are listed on screen.