SIGNAL / TOOLS

CWE-bench v2 results: can AI agents repair real security flaws?

SOURCECollinear AI / Hugging Face

PUBLISHEDNovember 3, 2026

LAYERTools

SIGNAL SCORE28/100

SXF VERIFIED BRIEF

Collinear AI released CWE-bench v2, a 100-task security-code repair evaluation with public programmatic graders and 11 tested models. The release provides reproducible testing material and cost-performance comparisons for agent-driven patches.

WHY IT MATTERS

An agent that names a flaw is different from an agent that repairs real code without breaking other behavior. Runnable verifiers allow more meaningful comparisons, but the same checks can overlook untested vulnerabilities and implementation details.

WHAT TO VERIFY

Pass@1 and pass@4 use different attempt budgets. The scores reflect a curated task set and particular harness. Automated verification cannot certify that an agent's patch is safe for production; independent tests and review remain necessary.

SXF RESEARCH EXPLAINER

What the technical evidence means.

Source-grounded analysis

TECHNICAL ANALYSIS / 01

What does CWE-bench v2 measure?

CWE-bench v2 gives a coding agent a checkout of an open-source repository and a concrete security issue to address. A submitted patch is judged by executable verification, not by how convincingly the agent explains its answer. This is a practical advance over tests that evaluate only multiple-choice knowledge of security categories.

The authors describe 100 tasks with human review and public graders. A replayable task makes it possible to inspect what changed and which test reported a success. Yet no published test suite covers every adjacent vulnerability, unforeseen input or production-specific setting.

Check primary-source evidence ↗

TECHNICAL ANALYSIS / 02

How to interpret model scores and costs

In the source's comparison, Gemini 4 Argon scores 67.8% pass@1 while Claude Opus 5.5 reaches 67.3%. With more attempts, both report higher pass@4 results. Because additional tries can improve the chance of eventual success, reports should state the attempt budget rather than promote the larger number without its conditions.

DeepSeek V4.1 Flash has a lower reported cost per run and a lower pass@1 score in the study. Real-world expense also includes tool calls, human audit, retries and consequences of missed regressions. Model choice should therefore depend on the actual patch workflow rather than a single percentage.

Check primary-source evidence ↗

TECHNICAL ANALYSIS / 03

A release checklist for agent-written security fixes

Test security patches using the benchmark's verifier, the application's wider regression suite, static analysis and deliberately adversarial inputs. Review the diff for permission handling, authentication paths, dependency updates and accidental data exposure. A different reviewer should confirm the underlying vulnerability has actually been addressed.

The lasting contribution of CWE-bench v2 is a public set of tasks that developers can rerun and challenge. It gives teams a better way to evaluate candidates, but not permission to bypass approvals for high-impact code changes.

Check primary-source evidence ↗

PRIMARY-SOURCE FACTS

What the evidence establishes.

Reviewed 2026-11-04

Task count

The authors publish 100 tasks derived from security problems in real open-source repositories. Primary evidence ↗

Grading

Tasks use runnable programmatic verifiers with human-reviewed specifications. Primary evidence ↗

Gemini comparison

Gemini 4 Argon reached a reported 67.8% pass@1 and 81.0% pass@4. Primary evidence ↗

Claude comparison

Claude Opus 5.5 is reported at 67.3% pass@1 and 78.0% pass@4. Primary evidence ↗

Cost example

DeepSeek V4.1 Flash is reported at 57.0% pass@1 and $0.04 per rollout. Primary evidence ↗

Combined task result

The authors say 95 tasks were solved by at least one tested model. Primary evidence ↗

Open research materials

A dataset is available through Hugging Face with applicable access terms. Primary evidence ↗

LIMITATIONS

All leaderboard numbers are benchmark-specific and author-reported. Deterministic tests can miss real exploit paths, and a passing patch may still be unsafe. A pass@4 figure cannot be compared directly with pass@1, and per-run costs vary outside the published setup.

PRACTICAL TAKEAWAYS

What readers can check next.

Verification checklist

Use equal attempt budgets when comparing benchmark results.
Inspect graders and verify what constitutes a security fix.
Run separate application security tests and human review.
Include retry and review costs before comparing deployed systems.

Read the related SXF guides

StateBench: a complementary evaluation of model reliability
Why exact problem conditions shape reported model outcomes.
Astra hardware design and validation
AI-generated artifacts still require independent checks.
AI agent security guide
Threat models, boundaries and safety tests.
Prompt injection guide
Review untrusted inputs to coding agents.
Transformers model integration contracts
Fast regression tests with explicit coverage limits.

TOPICSNo topic tag yet

MODELSClaude Opus 5.5 Gemini 4

CWE-bench v2 results: can AI agents repair real security flaws?

What the technical evidence means.

What does CWE-bench v2 measure?

How to interpret model scores and costs

A release checklist for agent-written security fixes

What the evidence establishes.

What readers can check next.

Verification checklist

Read the related SXF guides

Keep the context connected.

StateBench: Testing how language models track state changes over long documents

Transformers Model Integration Contracts

NVIDIA Kumo Relational Outperforms Supervised Machine Learning on Relational Data, No Training Required

How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications

Granite Speech 5.0: Transcribe, Translate, and Attribute more Accurately and Faster