Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Cyber LLM Benchmark Hub Logo
Cyber LLM Benchmark Hub
  • Home
  • Benchmarks
  • Contact
  • About
Support
Back to Benchmarks

CyberThreat-Eval

Threat Intelligence
Published March 10, 2026
View PaperView Code
Benchmark Overview

About this benchmark

Benchmark testing whether LLMs can automate a real-world threat-research workflow end to end, covering article triage and prioritization, deep search for related sources, and threat-intelligence drafting, scored with analyst-centric rather than text-similarity metrics

Dataset

3,830 samples

Expert-annotated benchmark spanning three workflow stages: triage and priority scoring over 488 articles, deep search across 55 articles, and TI drafting covering 1,310 IoCs, 1,565 MITRE ATT&CK TTPs, and 412 articles used for both threat-actor and root-cause narrative generation. Narrative output is graded by a six-dimension expert rubric (relevance, factual accuracy, comprehensiveness, clarity, coherence, source attribution) alongside latency and token cost.

7 tasks

Article TriagePriority ScoringDeep SearchIOC ExtractionTtp IdentificationThreat Actor GenerationRoot Cause Generation

Metrics

Results are ranked by the first listed metric; direction is shown for every metric below.

Triage precision (accepted) ↑Triage recall (accepted) ↑Priority-score pass rate ↑Avg. URLs yielding new information ↑IoC extraction precision ↑MITRE ATT&CK TTP precision ↑MITRE ATT&CK TTP recall ↑Threat-actor narrative accuracy (1-5) ↑Root-cause narrative accuracy (1-5) ↑
Model Results
Ranked by Triage precision (accepted) Β· higher is better
RankModelTriage precision (accepted)Triage recall (accepted)Priority-score pass rateAvg. URLs yielding new informationIoC extraction precisionMITRE ATT&CK TTP precisionMITRE ATT&CK TTP recallThreat-actor narrative accuracy (1-5)Root-cause narrative accuracy (1-5)Evaluated ByDateSource
1st
OpenAI o3-mini
o3-mini β€’ OpenAI
39.8%90.9%55.0%385.0%34.8%17.6%3.6564.627CyberThreat-Eval authorsMarch 10, 2026Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model
2nd
GPT-4o
gpt-4o β€’ OpenAI
35.9%99.0%56.6%3.5482.4%27.9%22.7%1.5283.458CyberThreat-Eval authorsMarch 10, 2026Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model
3rd
GPT-4o Mini
gpt-4o-mini-2024-07-18-ft-cyberthreat-eval β€’ OpenAI
34.8%98.0%59.0%0.2269.4%17.7%14.1%2.4053.819CyberThreat-Eval authorsMarch 10, 2026Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model
#4
GPT-4o
gpt-4o-2024-08-06-ft-cyberthreat-eval β€’ OpenAI
33.9%98.0%60.0%0.3888.5%23.9%18.5%3.6554.371CyberThreat-Eval authorsMarch 10, 2026Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model
Cyber LLM Benchmark Hub

Cyber LLM Benchmark Hub

Benchmarking frontier models across cybersecurity tasks.

BenchmarksContactAbout

Β© 2026 Cyber LLM Benchmark Hub