Pentesting made sense when releases happened every few months. A point-in-time assessment could provide an accurate picture of risk for weeks, sometimes months. Today, engineering teams ship continuously. Our State of AI in Pentesting survey of 200 CISOs and 200 engineering leaders, found that 76% deploy significant changes at least weekly, while nearly 40% deploy daily. Yet only 21% validate security on every release.
That gap has consequences.
- 79% are concerned about missing vulnerabilities between pentests.
- Half say findings are already outdated by the time the report arrives.
- 64% say security testing timelines influence release decisions, either delaying deployments or forcing teams to accept additional risk.
For many organisations, AI pentesting is becoming the practical way to close that gap. But because it's still a relatively new category, buyers often evaluate platforms using criteria that were designed for traditional penetration tests. Those criteria no longer tell you which platform will perform best once it's part of your development workflow.
Questions every AI pentesting vendor should be able to answer
Every vendor can demonstrate vulnerability discovery.
The harder questions appear once the platform becomes part of your engineering workflow. Can it make use of source code? How are AI agents prevented from leaving scope? Will the platform continue delivering value as your software changes?
Can the platform use source code?
Across more than 1,000 AI pentests, whitebox testing uncovered 7x more vulnerabilities while requiring fewer attempts than greybox testing alone.
Not every AI pentesting platform supports whitebox testing, and those that do don't necessarily use source code in the same way. Understanding how vendors use code, and the evidence behind the results, should be part of every evaluation.
Ask vendors:
- Does the platform support whitebox testing?
- Is code access optional?
- How is source code used during testing?
- How is customer source code protected?
- What evidence shows code access improves results?
Are you comparing vendors fairly?
A vendor evaluation is only useful if every platform is tested under comparable conditions.
If one vendor receives source code, another doesn't. If one platform runs for significantly longer, or consumes substantially more credits, the final reports become difficult to compare.
One of the recommendations in the guide is:
The checklist expands on this with practical questions covering authentication, source code, scope, validation, and reporting so every vendor is evaluated under the same conditions.
How is testing kept within scope?
AI agents are designed to explore applications. Buyers should understand exactly how those boundaries are enforced.
Ask how scope is enforced. Can production be excluded by default? Are domains allow-listed? What happens if an agent follows a redirect outside the agreed environment? Can tests be monitored or stopped while they're running?
The strongest platforms enforce these controls technically rather than relying on prompts or written instructions.
How does the platform keep pace with software changes?
Modern engineering teams don't stop shipping after a penetration test is complete. New code, new features, and new dependencies all change the attack surface.
Ask vendors how their platform fits into your release process. Can testing run automatically as software changes? How quickly can a new application be onboarded? Are fixes retested without scheduling another engagement? How long does it take to reach the first meaningful results?
These questions become increasingly important for organisations deploying multiple times a week or embedding security directly into CI/CD.
Common red flags during an evaluation
The guide also highlights warning signs that deserve a closer look.
- Vendors can't explain how findings are validated.
- Code access is dismissed without comparative evidence.
- Security guardrails rely on prompts instead of technical controls.
- Every demo focuses on familiar OWASP vulnerabilities with little evidence of business logic testing.
None of these automatically rule out a platform, but each deserves a closer look before making a decision.
Download the AI Pentesting Buyer's Guide
The examples above are just a small part of the evaluation framework.
The full guide also includes:
- A practical vendor evaluation checklist
- AI vs manual pentesting comparisons
- Research from more than 1,000 AI pentests
- Analysis of whitebox versus greybox testing
- Key findings from our State of AI in Pentesting report
- An anonymised customer case study where AI pentesting uncovered 13 issues after a 120-hour manual pentest reported none
Whether you're evaluating Aikido or another platform, the guide is designed to help you compare every vendor against the same set of criteria.
Download the full guide here:
