We burned 11.7 billion tokens to benchmark the cyber capabilities of 10 AI models with three attempts each, given 32 fresh off-the-shelf vulns to rediscover.
This benchmark is an evolution of our earlier known-CVE benchmark with a fresh, harder dataset, more models, and a closer look at what they find, how reliably they find it, and their quirks and trade-offs.
With this, we add GLM-5.3, DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731, Qwen3.8-Max, Kimi K3, and Grok 4.6 to the lineup.
TL;DR:
- DeepSeek V4 Pro 0813 finds the most vulnerabilities. Pooling three runs reaches 28 of 32 vulnerabilities.
- The most expensive model is not required. Three DeepSeek Pro runs cost about $295 and outperform Opus 5, Grok 4.6, or Sol. Three Flash runs cost $108 and reach 24 matching Grok’s best individual pass for less than a quarter of the cost.
- Models are inconsistent at recall; repetition remediates it. Single runs miss the breadth of findings, but pooling across runs fills this gap. DeepSeek Pro finds 17 vulnerabilities on its first pass but 28 across three.
- Open-source models now outperform the public frontier. DeepSeek V4 Pro topped every public closed model we tested on pooled vulnerability recall. Qwen, Kimi, and GLM-5.3 followed with strong consistency of findings without losing recall. Harnessed correctly, open models can now compete directly with the closed frontiers.
- The price of cheap coverage is noise. The open models caught up with the frontier at much cheaper rates but also produced the most false leads for the pipeline to reject.
How the benchmark works
We curated 32 vulnerabilities across real-world repositories and asked each model to rediscover them from the source.
Each model ran the complete set of 32 cases three times, for a total of 96 runs. This lets us measure how many vulnerabilities a model can find and how many it can find reliably.
The setup was a bounded version of the harness behind our AI Code Analysis. We replaced the model for the primary stage responsible for code exploration and reasoning with each of our contenders and gave the agent 30 turns max. The models also don’t have any internet access.
We were careful about dataset freshness. The 32 vulns were selected from recently disclosed CVEs to reduce the chance that the models had already seen the vulnerability, write-up or patch during training. The goal was to test vulnerability reasoning rather than memorization of well-known bugs. The cases, prompts, tools, and evaluation policy were all frozen across runs.
Some definitions to establish first, which we’ll use throughout this piece:
- Recall - How many vulnerabilities the model actually found
- Precision - How many of the findings reported were accepted
- Pass - One full run of a model on our 32-vulnerability dataset
Making model output variance work for us
The performance of LLMs depends largely on the problem at hand. Problems generally punish model output variance when you need reproducibility or reliability of the outcome.
Give an agent the same task in three fresh sessions, and it may solve it in three different approaches. None of the approaches necessarily has to be “wrong,” but the variation makes the system unpredictable, which is generally not good, as the same input, same system, different output.
However, vulnerability discovery is such a problem where you can actually make this property of LLMs work for you rather than against. Fundamentally, it is a search space problem, and search problems can benefit from this variance. In security audits, pooling the results of multiple runs can, and as you’ll see, has led to better outcomes.
Model inconsistency creates diversity, which can then potentially create better recall. If one run finds bugs that another misses, you get to keep both.
DeepSeek Pro found 17 vulnerabilities in its first pass. If that were the only run, that would be the final result.
But its second and third runs found vulnerabilities the earlier runs had missed. Pool all three together, and DeepSeek reached 28/32, taking the crown in our benchmark.
The same thing happened elsewhere. Qwen went from 19 in its first pass to 26 pooled. Kimi went from 17 to 25. Grok went from 24 to 26.
The exploratory models benefited enormously from repetition because different runs explored different useful paths. Grok gained much less because it was already finding roughly the same vulnerabilities each time.
17 cases were found by every model at least once. One defeated all ten. The remaining 14 are where the models stopped looking interchangeable.
DeepSeek had the highest pooled recall. Grok was the most consistent
DeepSeek found the most but not very consistently from one run to the next. Across three runs, V4 Pro 0813 eventually reached 28 of the 32 CVEs giving it the best overall result in the benchmark. But, only 10 of those were found in every run.
Grok was more conservative. It found slightly fewer vulns overall at 26/32 but when it did find something it was much more likely to find it again. It found 21 CVEs in all three runs, making it the most consistent model we tested surpassing all the frontiers.
Opus 5 also found 26 of 32 and recorded the strongest individual pass at 25 with 19 found consistently across three passes. Sol found 25/32 with the same 19-case consistency and produced some of the cleanest reports.
The DeepSeek models also produced far more candidate findings than the other models. That aggressive exploration helped its recall, but it also meant more false leads for the rest of the pipeline to filter out.
There’s a clear pattern, where the premium price of frontiers comes at reduced cost of post-run triage and better consistency.
Inside the investigation traces
DeepSeek Pro was the hungriest model in the benchmark. It used the most investigation budget with a median of 24 turns out of the 30-turn max limit we give to our agents.
Opus was the most relentless of all the closed frontiers. Investigations that returned no candidate took a median 29 turns but ones that produced a candidate took 19.
Sol averaged ~2.74 candidate issues per investigation and produced some of the cleanest reasoning findings we saw, usually keeping the path from attacker-controlled input to security impact easy to follow.
Qwen, phew. This model had CVE deja vu.
In 95 of 96 run traces, after recognizing the codebase, it spent turns contemplating on older, previously disclosed CVEs in the codebase from its training knowledge.
Those vulnerabilities were already patched in the version of the source we benchmarked on so most of this was wasted investigation budget before it eventually returned to the code reasoning.
Sometimes it’s deja vu served as a hint. Other times just sightseeing, but either way, Qwen was doing something much closer to autonomous variant analysis than plain pattern matching without being prompted for it.
Despite this quirk, it climbed the leaderboard by a lot, matching Opus and Grok at pass@3 pooled recall, sitting just behind DeepSeek’s lead.
Out of all, Kimi had the shortest exploration profile with a median of 12 turns. Its traces often explicitly converged, discarded weaker theories, and committed much earlier in the investigation than its peers.
It also had a 92.3% pooled precision, higher than every other open-weight model we tested. Lesser findings and lower false positives.
GLM closes the frontier gap
No, we did not forget about GLM-5.3. In fact, we stayed up all night for it (literally!).
Thanks to Z.AI, we had the opportunity to work closely with their team as an early evaluation partner for GLM-5.3, putting it through our cybersecurity benchmark and sharing feedback throughout the pre-release process.
During the evaluation, we tested two GLM-5.3 checkpoints, a pre-release preview checkpoint and the final released version.
GLM-5.3-preview beat GPT 5.6 Terra on cost-to-performance ratio at pass@3. It rediscovered 75% of the CVEs while costing 40% less and was also the most consistent of the open-weight models with 14/32 found across all three runs.
The final release of GLM improved upon this base and gained several points in recall and consistency while retaining roughly the same precision.
It was noticeably more persistent than the last version, which led to an increase in the median turn use from 18 to 23.
Overall the traces show broader and deeper exploration. Recall improved from 24/32 to 25/32 found, going head to head with GPT Sol while being 65.5% cheaper. It also surpassed GPT-5.6 Terra, GPT-5.6 Luna, DeepSeek Flash 0731 and finished just 1 CVE behind Opus 5 while still being 69.8% cheaper.
Its consistency also grew from 14/32 to 18/32 found consistently across all runs. These improvements came at the cost of cost per CVE found growing from $14.77 to $19.65. Still considerably cheaper than Sol, Opus, Grok while delivering competitive performance. It also had the highest pass@1 recall of all open models.
Open weights just beat the public frontier
In our last benchmark, the story was that open weights were catching up fast. This time they did more than catch up. Harnessed correctly, they operated on par and even beat the frontier.
DeepSeek delivered the best pooled coverage of any model ahead of Opus, Sol and Grok. GLM 5.3 caught up in consistency across runs and Qwen held its ground in pooled precision and recall.
The closed models still had their own edges, but the old hierarchy of one closed frontier model on top and everything else underneath did not survive this benchmark.
This benchmark tests the performance of only publicly available versions of the closed frontier models. We did not run Mythos or GPT 5.6 Daybreak class models in our tests.
Does this mean you can rip Sol/Opus out of your systems and drop-in any of the open-weights?
Yesn’t.
We mention “harnessed correctly,” and we mention it with stress. Open-weight models are formidable but not yet at the calibre to replace a frontier without a carefully designed harness.
As the frontier models grow better, we notice the trend that minimal harnessing == better performance. The open-weights gang is still a few steps from reaching this minimum.
So which model would we hire?
After ~11.7 billion tokens, “Which model is best?” is not the right question.
Better to ask where a model should fit in your system. Which researcher do you send in first, which one do you send in second, and who checks whose work?
You do not have to reach straight for the most expensive model. Three runs of DeepSeek V4 Pro 0813 cost about $295 and recovered 28 of 32 vulnerabilities, more than any single pass of Opus 5, Grok 4.6, or Sol, despite each of those passes costing roughly $450-$590.
Even three Flash runs cost just $108 and reach 24 matching Grok’s best single pass for less than a quarter of the price. Repeating an efficient model can buy more coverage than one expensive roll of the dice, but it comes with its own burdens. The cost is transferred to later stages of the pipeline.
Striking the balance between recall and precision is where harness engineering comes in.
We're excited by the updates! Worth sacrificing the sleep. Until next time!
.png)
This benchmark measures raw models. Aikido turns them into reliable vuln findings. Code Security Audit runs that reasoning across your whole source. Think of it as agentic AI SAST. Aikido Attack sends autonomous agents at your running app, pooling passes and discarding false leads the way the pipeline here rejected them. Harness the power of the latest models and secure your systems. See it at aikido.dev.
P.S. We're also hiring. Come build with us.
