For this research, we designed an identical set of prompts specifying the exact requirements and tech stack for three web applications. The prompts were intentionally security neutral - they provided no security-specific guidelines, and only focused on required functionality. Each of the five agents built all three applications, giving us 15 codebases to analyze. To find vulnerabilities, we combined static code analysis (CodeQL, Semgrep, Snyk), dynamic agent-based testing (Tenzai's hacking agent, Claude Code), and human verification of every finding. This produced a dataset of 70 vulnerabilities of varying severity across the 15 applications - from which we extracted a set of meaningful patterns and lessons. The good: coding agents appear to be quite effective at avoiding certain classes of bugs. A notable example were "notorious" categories of injection attacks. Across all the applications we developed, we didn't encounter a single exploitable SQLi or XSS vulnerability - two bug classes that have been staples of the OWASP Top 10 for years. Our observation is that coding agents perform well when the vulnerability class has well defined built-in protections. For SQL injection, agents consistently used parameterized queries, resulting in secure database interactions. With XSS, the agents' code often didn’t sanitize input, but it used frontend frameworks properly, which prevented vulnerabilities from becoming exploitable. The bad: While coding agents did relatively well with vulnerabilities that have clear and generic solutions - they struggled with issues that didn’t have one. Coding agents did very poorly in terms of properly enforcing authorization. They managed basic requirements reasonably well, but struggled significantly as authorization logic became more complex, despite clear and detailed guidance in our prompts. Another area of concern was business logic vulnerabilities. While human developers bring intuitive understanding that helps them grasp how workflows should operate, agents lack this “common sense” and depend mainly on explicit instructions. Without sufficiently detailed specification, agents can easily overlook important nuances. The ugly: The most concerning finding from this research wasn't the vulnerabilities in code the agents wrote, but ones that were introduced by code the agents didn't write. All the coding agents, across every test we performed, failed miserably when it came to security controls. It wasn’t that they implemented them incorrectly, in almost all cases - they didn’t even try. Across the board, critical controls like CSRF protection, security headers, and login rate limiting were nowhere to be found. Coding agents built what we explicitly asked for, often in reasonably secure ways, but completely failed to grasp "the bigger picture.” They lack the security mindset to proactively introduce defensive mechanisms that weren't explicitly requested.
After identifying these patterns, we set out to test different strategies to improve the security of the vibe coded apps. Using the same prompts, we evaluated three strategies: providing generic high-level security instructions in the prompt, having an agent analyze the prompt and inject security requirements before execution, and having the coding agent perform a security review of its own output. We'll show how these techniques solve a lot (but not all) of the issues we uncovered, and share actionable guidance for developers looking to ship AI-generated code more securely.
