
About this benchmark
Large-scale benchmark evaluating AI agents across the full vulnerability lifecycle on real open-source code: discovering the vulnerability, generating a crashing proof-of-concept, and producing a patch that holds without breaking the test suite
920 real-world vulnerabilities drawn from 139 widely used open-source projects, up from an initial 615-task set. Two settings: end-to-end, where the agent receives only the source code, and patch-only, where it also receives the crash log and PoC. Outputs are validated through four sequential stages (S1-S4), each requiring all previous stages to pass.
4 tasks
Results are ranked by the first listed metric; direction is shown for every metric below.
| Rank | Model | Patch-only success | End-to-end S1: PoC triggers crash | End-to-end S2: patch eliminates crash | End-to-end S3: tests still pass | End-to-end S4: matches ground-truth vulnerability | Evaluated By | Date | Source |
|---|---|---|---|---|---|---|---|---|---|
| 1st | GPT-5.4 gpt-5-4 • OpenAI | 87.1% | 67.9% | 66.2% | 65.9% | 22.2% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models |
| 2nd | Claude Opus 4.6 claude-opus-4-6 • Anthropic | 85.8% | 66.3% | 65.0% | 62.6% | 26.2% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models |
| 3rd | Claude Opus 4.6 claude-opus-4-6 • Anthropic | 84.1% | 39.7% | 39.5% | 37.9% | 15.7% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models |
| #4 | Gemini 3.1 Pro gemini-3-1-pro • Google | 83.0% | 47.4% | 44.3% | 43.8% | 20.5% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models |
| #5 | Claude Opus 4.5 claude-opus-4-5 • Anthropic | 82.3% | 24.9% | 21.9% | 19.2% | 7.6% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks |
| #6 | Gemini 3 Pro gemini-3-pro • Google | 77.6% | 29.6% | 23.6% | 22.6% | 5.0% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks |
| #7 | Claude 4.5 Sonnet claude-sonnet-4-5 • Anthropic | 77.4% | 18.1% | 12.1% | 10.6% | 3.4% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks |
| #8 | Claude 4.5 Sonnet claude-sonnet-4-5 • Anthropic | 68.9% | 9.3% | 7.2% | 5.4% | 2.3% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks |
| #9 | GPT-5.2 gpt-5.2-codex • OpenAI | 58.5% | 30.2% | 22.0% | 20.7% | 6.5% | CyberGym-E2E authors (UC Berkeley and collaborators) | July 17, 2026 | Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks |