AI Safety
Attacking the assistant: how LLM code tools can be steered into writing vulnerabilities
We catalogued the prompt shapes that steer LLM code assistants toward insecure output — and found that the defense that mattered was not the one everyone expected.
New software supply chains have old failure modes
AI code assistants are already inside millions of pull requests. Any systematic tendency to produce vulnerable code under adversarial pressure is therefore a supply-chain risk at industrial scale — not a demo curiosity.
Our taxonomy organises attack surface by workflow position: attacks inside the prompt, attacks through the repository context the assistant reads, and attacks through the test loop that an assistant uses to validate itself. The third one is the most interesting, and the most disturbing.
Defenses: fine-tuning vs detection
We evaluated five defenses across three assistant families. Fine-tuning on curated secure-output data shifted the failure rate but did not remove it; input sanitization caught the obvious shapes and missed the contextual ones, which know how to look like normal code.
The defense that moved the needle most was also the least glamorous: a semantics-aware scanner that checks generated code for the property the model was asked for, and flags drift. Assistant behavior is not something to trust at inference time; it is something to verify.
What we are publishing
The adversarial benchmark, the per-defense results and the audit protocol ship as the problem's deliverables. Assistants will keep getting better — we want the security bar to rise at the same pace.