The article focuses on the GTG-1002 campaign, where an AI agent executed various stages of a cyber intrusion. It highlights the capabilities and limitations of AI in offensive security operations. The case is significant as it marks the first documented instance of AI effectively managing tasks like reconnaissance, exploitation, and lateral movement within a single intrusion chain. This showcases how AI can enhance the efficiency of cyber attacks, raising concerns for defenders. Nico Waisman and Albert Ziegler, XBOW’s Head of Security and Head of AI, break down the strengths and weaknesses of large language models (LLMs) in this context. While LLMs can excel in structured tasks, they tend to struggle with planning drift, non-determinism, and state loss, leading to inaccuracies or "hallucinated" outcomes. The discussion emphasizes the need for a structured approach in utilizing AI for offensive security. XBOW presents its method for safely replicating AI-driven offensive strategies, emphasizing a governed architecture and validation process. This approach allows security teams to assess their vulnerabilities efficiently, cutting down the evaluation time from weeks to days. The article aims to equip defenders with a practical playbook to counter AI-driven threats effectively.