Cyber defenders face an overwhelming number of security alerts and evolving threats, highlighting the urgent need for AI systems to enhance security operations. Large Language Models (LLMs) have potential for automation in Security Operations Centers (SOCs), but current evaluations fall short. They don’t adequately assess how these models perform in real-world scenarios that matter to cyber defenders. This gap leaves AI developers without a clear benchmark to guide their work and users struggling to choose effective models. To tackle this issue, the authors introduce CyberSOCEval, a new suite of open-source benchmarks within the CyberSecEval 4 project. CyberSOCEval focuses on evaluating LLMs in two critical areas: Malware Analysis and Threat Intelligence Reasoning. These areas have been overlooked in existing security benchmarks. The evaluations show that larger, more advanced LLMs generally perform better, aligning with established training scaling laws. However, models designed for reasoning tasks do not exhibit the same improvements seen in other domains like coding or math. This indicates that these models are not yet equipped to handle cybersecurity analysis, which presents a significant opportunity for advancement. The findings also suggest that current LLMs have not yet reached their full potential in the context of cybersecurity. This indicates a considerable challenge for AI developers aiming to enhance the capabilities of AI in defending against cyber threats. The introduction of CyberSOCEval aims to drive improvements in both AI model development and the overall defense strategies employed by cybersecurity teams.