Elevating AI Security: The 15-Minute Threat Modeling Tactic

In the dynamic sphere of artificial intelligence, security professionals are facing new challenges. Adam Shostack, a notable threat-modeling expert, recently illustrated the urgency of addressing risks associated with AI technologies when a client reached out with concerns over an app that was handling customer data. The response time was tight, so Shostack allotted himself just 15 minutes to identify potential threats.

Within that brief window, he compiled a significant list of risks, including hallucination and bias, which traditional security frameworks like STRIDE would likely overlook. His approach was guided by a recently developed methodology known as PHANTOM-B. Rather than replacing conventional threat models, PHANTOM-B complements them by focusing on the unique challenges posed by large language models (LLMs).

Defining the PHANTOM-B Framework

PHANTOM-B serves to prompt important questions specific to AI components within a system. Each letter represents a unique threat: Prompt injection, Hallucination, Anthropomorphization, Non-explainability, Training issues, Overreliance, Missing security engineering, and Bias. Shostack’s framework allows security teams to conduct rapid assessments of AI systems, offering actionable insights without delving into exhaustive detail.

This rapid approach effectively addresses CISOs' dilemma: they’re pressed to secure AI technologies efficiently while grappling with existing security models that may not apply effectively to agile, AI-driven application architectures. Shostack emphasizes the need for “entry points” that can be integrated into short discussions or meetings, allowing for swift risk assessment and decision-making.

The Limitations of Traditional Threat Modeling

Traditional threat modeling processes frequently fall short, with organizations often struggling to apply them consistently. As development cycles speed up, there’s less time to engage in thorough risk assessments. Most companies save these modeling exercises for their most vital systems, which inadvertently leaves many applications exposed.

Jeff Williams, founder of OWASP, points out that AI hasn't necessarily disrupted threat modeling but rather highlighted existing shortcomings in the practice. Current methods often fail to keep pace with the rapid evolution of applications, often leading to outdated or incomplete risk models before they even begin. AI systems amplify these issues, as they operate in non-deterministic ways that are challenging to predict. Engineers can trace actions in conventional software, but AI interprets natural language and produces varied outputs for identical inputs, complicating the identification of risks.

Adapting Models for Non-Deterministic Systems

As Brian Glas from CODIFIC explains, the risk profiles for contemporary generative AI systems differ significantly from those of traditional applications. The blurred lines between data inputs and commands create new vulnerabilities, enabling potential failures to propagate across different systems. With product development moving rapidly, some problems may not surface until much later, potentially leading to significant security breaches.

Security experts, including those from NIST, have called for making threat modeling a continually active practice. While various guidelines have emerged to aid understanding of AI threats—including Microsoft's recommendations, the NIST AI Risk Management Framework, and OWASP's new Top 10 for LLM Applications—there is still no universally accepted framework for modeling risks in AI systems.

Maximizing Short Modeling Sessions

The concept of conducting impactful, condensed threat-modeling exercises raises questions about the feasibility of accomplishing tangible security work in such a short time. Shostack maintains that these sessions aren’t designed to provide exhaustive analyses; rather, they should surface meaningful risks that help inform decisions about system modifications or the need for deeper review.

Learning from agile practices, Shostack emphasizes that shorter modeling periods foster rapid iterations, allowing teams to reassess their findings swiftly without lengthy time losses. The aim is to gather concrete scenarios—stories about how the system could fail—to equip CISOs with information to make informed decisions about acceptable levels of risk and necessary security controls.

Integrating AI Considerations with Existing Frameworks

PHANTOM-B notably addresses peculiarities surrounding LLMs, such as anthropomorphization, challenging users or developers not to misjudge the model’s intent. This misplaced trust may influence system designs and determine an AI's level of authority.

Another issue is non-explainability. When LLMs are tasked with sorting applications, assessing images, or making business decisions, organizations need to justify their outcomes. However, asking a model for explanations doesn’t guarantee accuracy, as the responses may lack reproducibility. Understanding the "missing security engineering" issue is also essential, as adding LLMs doesn't mitigate traditional software vulnerabilities; it may amplify them instead.

Avoiding Common Pitfalls in Threat Modeling

As new AI tools proliferate, it's crucial for organizations to remember that they are still software applications reliant on frameworks that must address conventional security risks. Ignoring these fundamentals can leave organizations vulnerable to well-known threats. Williams cautions against rushing into analyzing AI-specific vulnerabilities while bypassing established security protocols.

CISOs should clarify how LLMs integrate into broader applications, ensuring that the necessary context surrounding trust boundaries and data flows is well-articulated. Attempting to catalogue all possible attacks is impractical; rather, teams should focus on protecting essential assets and utilizing existing preventive and detective controls. This approach allows for more effective modeling of potential attack scenarios to identify weaknesses and regulatory needs.

Ultimately, organizations must strive to adapt their threat modeling practices to better align with the developments in AI. By integrating frameworks that emphasize speed, adaptability, and foundational security principles, security teams can better protect their assets against the complex threats present in today's AI-driven landscape.