Renaissance LabsLearn

Models · Safety and alignment

Keeping AI agents safe: injection, incidents and design

Prompt injection and rules that limit damage

6 minBeginner8 chapters

An agent can act, so a wrong action has already happened. The film explains prompt injection, reviews real incidents, and presents two design rules, the lethal trifecta and Meta's rule of two. It argues that safety must be built into the surrounding system, and covers least privilege, sandboxes, separating trusted from untrusted input and human approval.

What to take away

  • A language model cannot reliably tell an instruction from you apart from an instruction hidden in content it reads, which is prompt injection.
  • The lethal trifecta is private data, untrusted content and a way to send information out; an agent should have at most two of three.
  • The film reports attacks on Anthropic's browser agent worked about one time in four without defenses and about one in nine with safeguards.
  • Safety cannot rely only on the model saying no, so least privilege, sandboxes and separating trusted from untrusted input matter.
  • Keep a human in the loop for anything irreversible, such as sending, paying, deleting or publishing, and keep backups.

Go deeper

Read the transcript77 lines, click any line to jump

This transcript comes from automatic captions and may contain small errors.

Keep watching