
About this benchmark
Benchmark testing whether LLMs can automate a real-world threat-research workflow end to end, covering article triage and prioritization, deep search for related sources, and threat-intelligence drafting, scored with analyst-centric rather than text-similarity metrics
Expert-annotated benchmark spanning three workflow stages: triage and priority scoring over 488 articles, deep search across 55 articles, and TI drafting covering 1,310 IoCs, 1,565 MITRE ATT&CK TTPs, and 412 articles used for both threat-actor and root-cause narrative generation. Narrative output is graded by a six-dimension expert rubric (relevance, factual accuracy, comprehensiveness, clarity, coherence, source attribution) alongside latency and token cost.
7 tasks
Results are ranked by the first listed metric; direction is shown for every metric below.
| Rank | Model | Triage precision (accepted) | Triage recall (accepted) | Priority-score pass rate | Avg. URLs yielding new information | IoC extraction precision | MITRE ATT&CK TTP precision | MITRE ATT&CK TTP recall | Threat-actor narrative accuracy (1-5) | Root-cause narrative accuracy (1-5) | Evaluated By | Date | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1st | OpenAI o3-mini o3-mini β’ OpenAI | 39.8% | 90.9% | 55.0% | 3 | 85.0% | 34.8% | 17.6% | 3.656 | 4.627 | CyberThreat-Eval authors | March 10, 2026 | Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model |
| 2nd | GPT-4o gpt-4o β’ OpenAI | 35.9% | 99.0% | 56.6% | 3.54 | 82.4% | 27.9% | 22.7% | 1.528 | 3.458 | CyberThreat-Eval authors | March 10, 2026 | Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model |
| 3rd | GPT-4o Mini gpt-4o-mini-2024-07-18-ft-cyberthreat-eval β’ OpenAI | 34.8% | 98.0% | 59.0% | 0.22 | 69.4% | 17.7% | 14.1% | 2.405 | 3.819 | CyberThreat-Eval authors | March 10, 2026 | Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model |
| #4 | GPT-4o gpt-4o-2024-08-06-ft-cyberthreat-eval β’ OpenAI | 33.9% | 98.0% | 60.0% | 0.38 | 88.5% | 23.9% | 18.5% | 3.655 | 4.371 | CyberThreat-Eval authors | March 10, 2026 | Table 4: Performance Metrics for Vanilla LLMs by Task Category, Task, Metric, and Model |