Learning Objectives — Threat Modeling AI Systems

By the end of this lesson, learners will be able to:

• Define threat modeling and its role in AI security.
• Explain the objectives of threat modeling within AI environments.
• Identify common adversaries that target AI systems.
• Recognize attack surfaces across the AI lifecycle.
• Understand how AI threats differ from traditional cybersecurity threats.
• Describe the relationship between assets, threats, vulnerabilities, and controls.
• Apply threat modeling concepts to machine learning pipelines.
• Evaluate risks affecting training, deployment, and inference environments.
• Understand how threat models support security-by-design practices.
• Apply threat modeling concepts to certification exam scenarios.

Key Concepts — Threat Modeling AI Systems

• Threat Modeling
• Security-by-Design
• Attack Surface
• Threat Actor
• Adversary
• Risk Assessment
• Vulnerability
• Asset Identification
• Trust Boundary
• Attack Tree
• STRIDE
• AI Attack Surface
• Training Pipeline
• Data Poisoning
• Model Poisoning
• Model Theft
• Prompt Injection
• Inference Security
• Supply Chain Risk
• Model Extraction
• Insider Threat
• Security Architecture
• Mitigation Strategy
• Risk Prioritization
• Defense-in-Depth

Transcript — Threat Modeling AI Systems

Welcome to Lesson 3.1: Threat Modeling AI Systems.

As we begin Module 3, our focus shifts from governance and data protection into the engineering and architectural foundations of AI security.

So far, we’ve discussed risk management, governance frameworks, privacy engineering, compliance requirements, and data security controls. These topics help organizations understand and manage risk at a strategic level.

Now we move into the technical side of AI security.

And one of the most important activities in secure engineering is threat modeling.

Before organizations can secure AI systems, they must first understand how those systems can be attacked.

Security teams often face a simple but important question:

What are we defending against?

Threat modeling helps answer that question.

It provides a structured process for identifying potential adversaries, understanding attack paths, evaluating vulnerabilities, and selecting appropriate security controls.

Threat modeling is widely used throughout cybersecurity, but AI systems introduce unique challenges that require additional consideration.

Unlike traditional applications, AI systems rely on data, models, training pipelines, inference services, and complex dependencies.

Each of these components creates opportunities for attackers.

Threat modeling helps organizations identify those opportunities before adversaries do.

Let’s begin with a basic definition.

Threat modeling is the process of identifying assets, threats, vulnerabilities, attack paths, and security controls within a system.

The objective is not to predict every possible attack.

That would be impossible.

Instead, the goal is to systematically evaluate how a system could be compromised and determine what protections are necessary to reduce risk.

Threat modeling supports proactive security.

Rather than waiting for incidents to occur, organizations identify risks during design and development.

This approach aligns closely with the concept of security-by-design.

Security-by-design means incorporating security considerations into architecture and engineering decisions from the beginning.

Threat modeling is one of the primary tools used to achieve that objective.

To understand threat modeling, we must first understand several key concepts.

The first concept is assets.

Assets are anything that has value and requires protection.

Within AI systems, assets can take many forms.

Training datasets.

Model weights.

Feature stores.

Inference APIs.

Prompt libraries.

Embeddings.

System credentials.

Infrastructure.

And intellectual property.

Each of these assets may attract attackers for different reasons.

For example, proprietary model weights may represent years of research and significant financial investment.

Customer data may contain sensitive information protected by privacy regulations.

Inference APIs may provide access to valuable capabilities.

Before security teams can protect assets, they must identify them.

Asset identification is therefore one of the first steps in threat modeling.

The second concept is threats.

A threat represents a potential event or action that could cause harm.

Threats may originate from external attackers, malicious insiders, competitors, criminal organizations, nation-state actors, or even accidental human error.

Different adversaries have different objectives.

Some seek financial gain.

Others seek intellectual property.

Others may attempt to disrupt operations or damage reputations.

Understanding adversary motivations helps organizations prioritize defensive efforts.

For example, an AI startup may be particularly concerned about model theft.

A healthcare provider may prioritize patient privacy.

A financial institution may focus on fraud and data integrity.

Threat modeling helps align security controls with likely attack scenarios.

The next concept is vulnerabilities.

A vulnerability is a weakness that can be exploited by a threat actor.

Vulnerabilities may exist within software, infrastructure, processes, configurations, or AI-specific components.

Examples include weak access controls, exposed APIs, insecure training environments, insufficient monitoring, or unvalidated data sources.

Threats exploit vulnerabilities to impact assets.

This relationship is fundamental to risk analysis.

No vulnerability does not necessarily mean no risk.

But vulnerabilities significantly increase the likelihood of successful attacks.

Now let’s examine attack surfaces.

The attack surface represents all possible points where an attacker may interact with a system.

Traditional applications typically have attack surfaces involving applications, databases, APIs, users, and infrastructure.

AI systems introduce additional attack surfaces.

Training pipelines.

Feature engineering processes.

Model repositories.

Inference endpoints.

Vector databases.

Prompt management systems.

Third-party datasets.

And machine learning infrastructure.

Every component that interacts with data or models becomes part of the attack surface.

As AI ecosystems become more complex, attack surfaces often expand significantly.

This is one reason why threat modeling is particularly important for AI environments.

Let’s explore how threat modeling is performed.

Most threat modeling exercises begin by understanding system architecture.

Teams create diagrams showing major components, data flows, users, external dependencies, and trust boundaries.

Trust boundaries are particularly important.

A trust boundary represents a transition point where different security assumptions apply.

For example, information moving from a public network into a private environment crosses a trust boundary.

Similarly, data entering a training pipeline from an external source crosses a trust boundary.

Threats often emerge where trust boundaries exist.

Identifying these boundaries helps security teams focus their analysis.

Once the architecture is understood, teams identify potential threats.

Several methodologies support this process.

One widely used framework is STRIDE.

STRIDE categorizes threats into six groups:

Spoofing.

Tampering.

Repudiation.

Information Disclosure.

Denial of Service.

And Elevation of Privilege.

Although originally developed for traditional software systems, STRIDE can be adapted for AI environments.

For example, model tampering may involve modifying training data or model parameters.

Information disclosure may involve model inversion attacks.

Denial of service may target inference infrastructure.

Threat modeling frameworks provide structured ways to evaluate risk systematically.

Another useful technique is attack tree analysis.

An attack tree visualizes how attackers might achieve a particular objective.

The root of the tree represents the attacker’s goal.

Branches represent alternative attack paths.

For example, an attacker seeking model theft may target repositories, inference endpoints, insider access, or backup systems.

Attack trees help teams understand multiple attack scenarios and prioritize defensive measures.

Now let’s examine AI-specific threats.

One of the most significant threats is data poisoning.

Data poisoning occurs when attackers manipulate training data to influence model behavior.

By inserting malicious records into datasets, attackers can degrade performance or create targeted vulnerabilities.

Model poisoning represents a related threat.

Rather than manipulating data, attackers directly influence model updates or training processes.

These attacks are particularly relevant within federated learning environments.

Another major concern is model theft.

AI models often represent significant intellectual property.

Attackers may attempt to steal model weights, architectures, training methodologies, or proprietary capabilities.

Model extraction attacks are one common technique.

By repeatedly querying a model, attackers attempt to reconstruct its behavior.

Over time, they may develop a functional approximation of the original model.

Prompt injection has emerged as another important threat within generative AI systems.

Prompt injection occurs when attackers manipulate model inputs to influence outputs or bypass controls.

These attacks can undermine guardrails, expose sensitive information, or alter system behavior.

Inference systems are particularly vulnerable to prompt-based attacks.

This highlights why AI threat modeling must account for emerging attack techniques.

Supply chain threats are also increasingly important.

Modern AI systems depend on external datasets, pretrained models, open-source libraries, cloud services, and third-party integrations.

Each dependency introduces potential risk.

Compromised components may introduce vulnerabilities throughout the AI lifecycle.

Threat modeling helps organizations identify and evaluate these dependencies.

Insider threats deserve attention as well.

Not all threats originate externally.

Employees, contractors, or partners with authorized access may intentionally or accidentally create security risks.

Access management, monitoring, and governance controls help reduce insider risk exposure.

After identifying threats, organizations evaluate mitigations.

Mitigations reduce likelihood, impact, or both.

Examples include:

Access controls.

Encryption.

Monitoring.

Model validation.

Security testing.

Network segmentation.

Secure development practices.

And incident response planning.

Effective threat models connect identified threats with appropriate controls.

This creates traceability between risks and defensive measures.

Threat modeling is not a one-time exercise.

AI systems evolve continuously.

Models are retrained.

Datasets change.

Infrastructure changes.

New attack techniques emerge.

Threat models should therefore be reviewed regularly.

Many organizations update threat models during architecture changes, major releases, or security reviews.

Continuous improvement helps maintain relevance as systems evolve.

Let’s consider a practical example.

Imagine an organization developing an AI-powered financial assistant.

The system includes training data, model repositories, inference APIs, cloud infrastructure, and customer-facing applications.

Threat modeling identifies several risks.

Data poisoning during training.

Model extraction through public APIs.

Credential theft affecting administrators.

Prompt injection targeting customer interactions.

Third-party dependency compromise.

Security teams evaluate each threat, identify mitigations, assign ownership, and prioritize implementation efforts.

As a result, security becomes integrated into architecture decisions rather than added after deployment.

This is the true value of threat modeling.

For certification exams, remember several key concepts.

Threat modeling identifies assets, threats, vulnerabilities, attack surfaces, and controls.

Asset identification is the foundation of analysis.

Trust boundaries help identify areas of elevated risk.

STRIDE provides a structured threat categorization framework.

Attack trees visualize attack paths.

AI-specific threats include data poisoning, model poisoning, model theft, model extraction, prompt injection, and supply chain compromise.

Threat modeling supports security-by-design and proactive risk management.

And threat models should evolve alongside AI systems.

To summarize, threat modeling is one of the most important practices within secure AI engineering.

It helps organizations understand how systems can be attacked, prioritize defensive measures, and build security into architecture decisions from the beginning.

As AI systems become increasingly valuable and complex, threat modeling provides a structured foundation for identifying and managing risk.

In the next lesson, we’ll explore Secure Feature Engineering and examine how organizations protect one of the most influential stages in the machine learning lifecycle: the transformation of raw data into model-ready features.