AI security engineering

Securing an AI application is ordinary security engineering plus one new fact: a language model cannot reliably tell the instructions it should follow from the text it was merely asked to read. No patch fixes that. Everything distinctive in this course follows from it, especially the working assumption that the model will sometimes be manipulated and that the system around it must make that survivable.

The course starts at “why is this different from web security?” and ends with a complete security design for a multi-tenant agent platform. It assumes you can read Python and know what an HTTP API and an access token are. It does not assume a security background.

The stages#

Security work attaches to the stages of an application’s life, and the course is organized the same way: attacks first, then the controls for each stage.

flowchart LR
  D["<b>Design</b><br/><small>threat model</small>"] --> DA["<b>Data</b><br/><small>provenance, privacy</small>"]
  DA --> MO["<b>Model</b><br/><small>supply chain, signing</small>"]
  MO --> B["<b>Build</b><br/><small>red team, eval gates</small>"]
  B --> DE["<b>Deploy</b><br/><small>isolation, secrets</small>"]
  DE --> R["<b>Runtime</b><br/><small>guardrails, policy</small>"]
  R --> A["<b>Agents</b><br/><small>identity, sandboxes</small>"]
  A --> O["<b>Operate</b><br/><small>detect, respond</small>"]
  O -.->|"incidents become tests"| D
  class D neutral
  class DA,MO memory
  class B queue
  class DE compute
  class R,A warn
  class O io
classDef neutral fill:#faf9f5,stroke:#5f5e55,color:#141413,stroke-width:1.5px
classDef compute fill:#e1e9f7,stroke:#2443b5,color:#141413,stroke-width:1.5px
classDef memory fill:#e1e9f7,stroke:#2443b5,color:#141413,stroke-width:1.5px
classDef queue fill:#e1e9f7,stroke:#2443b5,color:#141413,stroke-width:1.5px
classDef io fill:#e1e9f7,stroke:#2443b5,color:#141413,stroke-width:1.5px
classDef default fill:#e1e9f7,stroke:#2443b5,color:#141413,stroke-width:1.5px
classDef warn fill:#f9e2e9,stroke:#b3154f,color:#141413,stroke-width:2px,stroke-dasharray:5 3
classDef good fill:#e2efd8,stroke:#2b6a24,color:#141413,stroke-width:2px

How each lesson works#

Every lesson has the same shape, so you always know where to look:

1. The idea in one minute. The claim, before the detail.
2. A picture of the attack or the defense.
3. How it really works. The mechanism, precisely, with the tools named.
4. Code, where a mechanism is worth running: a small Python program using only the standard library. Attacks run against simulated models inside the program; nothing targets a real system.
5. Where it breaks. The limits of the control and what it does not prove.
6. Remember this, Try it, and Check yourself, with answers you can reveal after you have tried.

Attacks come before defenses, because a defense makes sense only once you can see what it stops, and what it does not.

What you need#

• Python 3.10 or newer and a terminal. Every program runs offline.
• No GPU, no API key, no lab environment.

Names, versions, and dates#

Framework editions, incident references, specification versions, and legal dates change. Lessons that depend on them list their sources and the date they were checked. The principles underneath (untrusted input, least privilege, isolation, defense in depth) are far older than any model and will outlast the current ones.