# CWE-bench v2 open-sources 100-task security benchmark for AI agents on Hugging Face

> **Key takeaway:** Collinear AI released CWE-bench v2 as an open, reproducible benchmark where AI agents must find and fix real security vulnerabilities across 91 CWE classes in Dockerized environments without breaking functionality.

**Published:** 2026-11-03T18:02:28.941859+00:00 
**Source:** Hugging Face Blog & Announcements 
**Category:** ai-open-models 
**Canonical URL:** https://fosswire.org/news/cwe-bench-v2-open-sources-100-task-security-benchmark-for-ai.html 

## Summary
Collinear AI has published CWE-bench v2, a public dataset of 100 security evaluation tasks hosted on Hugging Face, designed to measure how well AI agents can identify and patch real vulnerabilities in open-source codebases. Each task provides a checkout of an actual repository, a realistic instruction, and a deterministic programmatic verifier that checks whether the fix works without regressing existing functionality. The benchmark covers 91 distinct Common Weakness Enumeration (CWE) categories, and every task was manually reviewed against six quality criteria including resistance to memorized upstream patches.

The release includes an open grader and harness, with results tracked on two Hugging Face Hub leaderboards: one for open-weight models updated via community pull requests, and another Space comparing API-only models including Gemini, Claude, and GPT. Of 11 models tested so far, Gemini 4 Argon leads at 67.8% pass@1, though even the top performer fails on nearly one in five tasks. The benchmark can be run directly via the Harbor tool without local cloning, and the tasks are tagged as rl-environments for training defensive agents.

## Benchmark structure and evaluation methodology
- **Dockerized task environment:** Each task is a self-contained Harbor task with a Dockerized runtime, an instruction, and a deterministic verifier, enabling reproducible evaluation without manual environment setup.
- **Programmatic scoring with LLM-judge curation:** All verification checks are fully automated and deterministic, shaped by LLM judge panels toward hardness and fairness rather than pattern-matching on fix structure.
- **Coverage breadth:** The 100 public tasks span 91 distinct CWE classes, requiring models to generalize across vulnerability types rather than recognize familiar patterns.
- **Anti-memorization design:** Verifiers are constructed to fail on known upstream patches, ensuring that models must generate novel fixes rather than reproduce memorized solutions.
- **Held-out evaluation set:** A separate private task set is maintained for official progress tracking; access requires contacting Collinear AI, preventing training set contamination.

## Why it matters
CWE-bench v2 provides the open-source AI security community with a reproducible, cost-transparent benchmark for evaluating autonomous code repair capabilities, addressing a gap where proprietary evaluations have dominated. By hosting datasets and leaderboards on Hugging Face with community-driven result submission via pull requests, it reduces vendor lock-in in AI safety evaluation and enables independent researchers to compare open-weight models against API-only services on equal footing. The rl-environment tagging also positions these tasks as training infrastructure for defensive security agents, potentially accelerating open research into automated vulnerability remediation.