systemlayers · AI · Security · Research
Taking AI systems apart,
layer by layer
I write about AI security: how models and agents break, how they get attacked, and what actually works to defend them. No hype, just research, real-world lessons, and things you can reproduce yourself.
All posts
- Who decides what exists: the perception layer
Alignment and prompt engineering focus on what a model decides to do. This article asks about an earlier security boundary: how the model reconstructs the objects, boundaries and roles it reasons over, and why nothing in the current stack owns that step.
- Human in the wrong loop (HITWL): when human oversight is a signature, not a control
Human oversight is the standard safety answer for agentic systems, but a gate placed where the reviewer's only evidence comes from the process being reviewed is not a control, it is a signature. Human in the wrong loop is informational, not behavioral: the reviewer can be expert and unhurried and still be unable to see the failure. A single spreadsheet case shows a vendor's own remedy, review the output, sitting exactly in the wrong loop.
- Structural desynchronization in Google Sheets and Microsoft 365 Copilot: a single adversarial cell fabricates rows, charts, and a new sheet
One crafted cell in an otherwise valid CSV makes Gemini in Google Sheets and Microsoft 365 Copilot report, chart, and even write out records that do not exist in the file. Five rows are read as fourteen, thirty as thirty-nine, and in Excel Copilot the fabricated rows are materialized into a new sheet. The extra records are attacker-controlled, the mechanism is structural desynchronization rather than instruction injection, and both vendors classified it as a non-security or by-design issue.
- Five data structures, one failure: what structural desynchronization does inside an autonomous agent flow
An agent reads tool output, decides, and acts. The step where it decides what counts as one record is not deterministic. Across JSON, XML, CSV, raw nginx logs and ripgrep output, four frontier models reconstruct entity boundaries that diverge from the input, and five of ten configurations diverge in 100% of runs at both T=0 and T=1. Six experiments across five input formats, with the raw inputs and outputs.
- Structural desynchronization in Gmail and Gemini: a fabricated inbox and real email content exfiltrated from Gmail into a shared Calendar
A single crafted email makes Gemini reconstruct fabricated inbox entries as real, then an embedded directive drives an autonomous Calendar event that exfiltrates content from a genuine inbox message into a shared Calendar event visible to colleagues. It arrives looking like indirect prompt injection, but the mechanism underneath both effects is structural desynchronization, and the fabrication half involves no instruction at all.
- Structural desynchronization: one field, ten models, six records
Two records go into ten independently built language models. All ten return six. The extra records are written by whoever controls a single field, and one of the two real records does not survive reconstruction as a unit. No instruction, no jailbreak, no override.