Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Back to Benchmarks

CyberGym-E2E

Vulnerability Analysis
Published June 3, 2026
View PaperView CodeView Dataset
Benchmark Overview

About this benchmark

Large-scale benchmark evaluating AI agents across the full vulnerability lifecycle on real open-source code: discovering the vulnerability, generating a crashing proof-of-concept, and producing a patch that holds without breaking the test suite

Dataset

920 samples

920 real-world vulnerabilities drawn from 139 widely used open-source projects, up from an initial 615-task set. Two settings: end-to-end, where the agent receives only the source code, and patch-only, where it also receives the crash log and PoC. Outputs are validated through four sequential stages (S1-S4), each requiring all previous stages to pass.

4 tasks

Vulnerability DiscoveryPoc GenerationPatch GenerationPatch Validation

Metrics

Results are ranked by the first listed metric; direction is shown for every metric below.

Patch-only success ↑End-to-end S1: PoC triggers crash ↑End-to-end S2: patch eliminates crash ↑End-to-end S3: tests still pass ↑End-to-end S4: matches ground-truth vulnerability ↑
Model Results
Ranked by Patch-only success · higher is better
RankModelPatch-only successEnd-to-end S1: PoC triggers crashEnd-to-end S2: patch eliminates crashEnd-to-end S3: tests still passEnd-to-end S4: matches ground-truth vulnerabilityEvaluated ByDateSource
1st
GPT-5.4
gpt-5-4 • OpenAI
87.1%67.9%66.2%65.9%22.2%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models
2nd
Claude Opus 4.6
claude-opus-4-6 • Anthropic
85.8%66.3%65.0%62.6%26.2%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models
3rd
Claude Opus 4.6
claude-opus-4-6 • Anthropic
84.1%39.7%39.5%37.9%15.7%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models
#4
Gemini 3.1 Pro
gemini-3-1-pro • Google
83.0%47.4%44.3%43.8%20.5%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 4: Success rates (%) on the expanded 920-task benchmark with newer frontier models
#5
Claude Opus 4.5
claude-opus-4-5 • Anthropic
82.3%24.9%21.9%19.2%7.6%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks
#6
Gemini 3 Pro
gemini-3-pro • Google
77.6%29.6%23.6%22.6%5.0%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks
#7
Claude 4.5 Sonnet
claude-sonnet-4-5 • Anthropic
77.4%18.1%12.1%10.6%3.4%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks
#8
Claude 4.5 Sonnet
claude-sonnet-4-5 • Anthropic
68.9%9.3%7.2%5.4%2.3%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks
#9
GPT-5.2
gpt-5.2-codex • OpenAI
58.5%30.2%22.0%20.7%6.5%CyberGym-E2E authors (UC Berkeley and collaborators)July 17, 2026Table 3: Success rates (%) across model and harness combinations on the initial 615 tasks
Cyber LLM Benchmark Hub

Cyber LLM Benchmark Hub

Benchmarking frontier models across cybersecurity tasks.

BenchmarksContactAbout

© 2026 Cyber LLM Benchmark Hub