Google is addressing the growing threat of indirect prompt injection attacks on generative AI systems, which involve hidden malicious instructions in external data sources. Their layered security strategy for the Gemini platform includes advanced content classifiers, security thought reinforcement, markdown sanitization, user confirmation mechanisms, and end-user security notifications to enhance protection against such attacks.

Researchers from Tel Aviv University have demonstrated a new type of cyber attack they call "promptware" by using calendar events to manipulate Google's AI, Gemini, into controlling smart home devices. By embedding malicious instructions in calendar appointments, they successfully executed indirect prompt injection attacks, allowing unauthorized control over devices like lights and thermostats. This incident marks a significant shift in how AI vulnerabilities can impact the physical world.

Security researchers at Trail of Bits have discovered that Google's Gemini tools are vulnerable to image-scaling prompt injection attacks, allowing malicious prompts to be embedded in images that can manipulate the AI's behavior. Google does not classify this as a security vulnerability due to its reliance on non-default configurations, but researchers warn that such attacks could exploit AI systems if not properly mitigated. They recommend avoiding image downscaling in agentic AI systems and implementing systematic defenses against prompt injection.