Threat model an LLM feature before you give it tools

To threat model an LLM feature, map where outside text enters the prompt and where the model’s output goes next. Run STRIDE over each crossing, check it against the OWASP Top 10 for LLM Applications 2025, and put authorisation in the tools. A prompt can’t be the guard, because OWASP says it’s unclear whether prompt injection has any fool-proof prevention.

The guidance below comes from the OWASP Top 10 for LLM Applications 2025 and OWASP’s Threat Modeling Cheat Sheet, last verified on 11 October 2026. The example feature is made up, and none of it was tested for this post.

This is part 20 of the series. Part 7 covered prompt injection against a coding agent, and Part 6 the agent’s own permissions. This part is about an LLM feature you ship to your own users.

Four questions and STRIDE give you the method

The Threat Modeling Manifesto builds threat modelling on four questions:

1. “What are we working on?”
2. “What can go wrong?”
3. “What are we going to do about it?”
4. “Did we do a good enough job?”

OWASP’s Threat Modeling Cheat Sheet turns them into four steps: model the system, find the threats, decide a response, review.

For the first step, the cheat sheet wants “a clear view of trust boundaries, data flows, data stores, processes, and the external entities which may interact with the system”. A trust boundary is any line where data passes between parts you trust differently, like a customer’s email and your server.

For the second step, it uses STRIDE, a set of six prompts first developed by Microsoft employees. The six are spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privileges. The cheat sheet’s advice is to “Start with STRIDE for a technical design”. An LLM feature is a technical design, so STRIDE fits it as it is.

A made-up example: a reply assistant in a helpdesk app

The reply assistant is made up for this post. A helpdesk app adds a “Draft reply” button to every customer ticket. A staff member clicks it, and an LLM reads the ticket, searches the help centre, looks up the customer’s order and drafts an answer. It can call two tools, lookup_order and propose_refund. The staff member edits the draft and sends it by email.

The assets come first, because they decide how bad each threat is:

A rough data flow for the reply assistant, as a text diagram:

Customer email ----> Ticket store ---------------------+
                                                       |
Staff browser --"Draft reply"--> App server <----------+
                                   |   ^   ^
                                   |   |   +-- Help centre search
                                   v   |
                             Model provider
                                   |
                                   v
                 Tool calls: lookup_order, propose_refund
                                   |
                                   v
               Orders DB, refund service
                                   |
App server --draft--> Staff browser --send--> Email to customer

Five crossings in that diagram are trust boundaries. At most of them, text arrives that nobody on your team wrote.

Five trust boundaries in the reply assistant

The trust boundaries in an LLM feature are where untrusted text reaches the model. They are also where the model’s output reaches something that acts on it:

The attacker in this example never logs in. They send an email, and the staff member’s click puts that email in front of a model that can call tools. OWASP’s LLM01 entry calls this indirect prompt injection: instructions hidden in external content the model reads.

A tool result is input too. The delivery note on an order is free text a customer typed, and lookup_order hands it straight back to the model.

The model provider is a boundary as well. Everything in the prompt leaves your network, so send the fields a reply needs and read the provider’s data retention terms.

Retrieval has its own trap. The help centre holds internal articles, and the draft goes to a customer. OWASP’s LLM08 entry asks for “permission-aware vector and embedding stores”, so the search should filter by audience before the model sees anything.

STRIDE over the reply assistant, mapped to OWASP 2025

STRIDE gives one row per kind of threat. Each row below maps a threat in the made-up reply assistant to an OWASP 2025 entry and a control:

The rows are for the made-up example. The OWASP entries were last verified on 11 October 2026.

Every row needs a response. The cheat sheet’s response list is mitigate, eliminate, transfer or accept. Eliminate means “Simply remove the feature or component that is causing the threat.” For a tool that does too much, eliminate can be the cheapest answer. Ship version one without propose_refund, and the elevation row goes away.

Enforce authorisation in the tool and the data layer, never in the prompt

Authorisation for an LLM feature has to sit in code the model can’t argue with. OWASP’s LLM06 Excessive Agency entry calls this complete mediation: “Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not.” Its LLM07 entry adds that checks like these “must not be delegated to the LLM, either through the system prompt or otherwise.”

An example lookup_order handler, written for the made-up feature, puts the limit in the database query:

// The model chooses the order number. The server decides whose orders it can see.
export async function lookupOrder(
  args: { orderNumber: string },
  ctx: ToolContext,
) {
  const order = await db.orders.findFirst({
    where: {
      number: args.orderNumber,
      customerId: ctx.ticket.customerId, // from the ticket record, never from model text
      tenantId: ctx.session.tenantId, // from the signed-in session
    },
    // Return the fields a reply needs, nothing more.
    select: { number: true, status: true, shippedAt: true, total: true },
  })
  return order ?? { error: 'No order with that number on this ticket.' }
}

The database side matters as much. OWASP’s LLM06 page describes a read tool whose database identity “not only has SELECT permissions, but also UPDATE, INSERT and DELETE permissions”. Give lookup_order a database role that can read orders and nothing else.

propose_refund gets the same treatment. It creates a pending proposal scoped to the ticket’s customer, and nothing moves until a staff member approves it on screen. The refund service then checks that staff member’s own refund limit. LLM06 says approval “may be implemented in a downstream system (outside the scope of the LLM application) or within the LLM extension itself”. Either way, the model’s output never counts as the approval.

The cheat sheet’s review step asks: “Can the agreed upon mitigations be tested?” For these two tools, yes, with no model in the loop. Call lookupOrder with another customer’s order number and expect nothing back.

Model output is untrusted input for the browser and the email

Model output needs the same handling as any user input. The LLM05 Improper Output Handling entry says: “Treat the model as any other user, adopting a zero-trust approach”. It lists Markdown or JavaScript from the model, interpreted by the browser, as a route to cross-site scripting (XSS).

Images are the quieter leak. The LLM01 entry describes hidden instructions in a web page that make a model insert an image whose URL leaks the private conversation. A browser fetches an image as soon as it renders it. Whatever text sits in that URL reaches someone else’s server, and nobody had to click.

The reply assistant handles its output with these controls:

• Render the draft as plain text, or as a Markdown subset with no images and no raw HTML.
• Set a Content Security Policy img-src that allows images from your own hosts, and nowhere else.
• Show every link’s full address in the draft, and flag links to domains you don’t know before the staff member sends it.
• Escape the draft for the email template, which LLM05 lists as its own route to phishing.

The system prompt belongs in the same picture. OWASP’s LLM07 entry says “the system prompt should not be considered a secret, nor should it be used as a security control.” Keep API keys, connection strings, role rules and refund limits out of it. If reading it would help an attacker, the information is in the wrong place.

Two prompts to draft and check the threat model

A coding agent can do the reading for a first draft of the threat model. Neither prompt depends on a tool-specific feature, and neither asks for attack payloads.

Prompt 1: draft the threat model from the code

An example prompt for when the LLM feature exists in code, even as an early branch.

Task: draft a threat model for <the LLM feature> in this repository.

Read only the code this needs: where the prompt is built, what goes
into it, which tools the model can call, and where its output goes.

1. List the assets the feature can reach: data, money, actions, secrets.
2. List every input to the prompt and who can write it: the user,
   other users, documents, retrieved content, tool results, websites.
3. List every place the model's output goes: browser, email, database,
   tools, other services.
4. Mark each trust boundary those inputs and outputs cross.
5. For each boundary, go through STRIDE and name the threats that
   apply. Map each one to an OWASP Top 10 for LLM Applications 2025
   entry.
6. For each tool, say where authorisation is enforced. Flag any check
   that exists only in the prompt or the system prompt.

Cite the file and line for every claim. Mark anything you could not
confirm in the code as unverified.
Do not edit files. Do not write or run attack payloads.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

A check flagged in step 6 is the first thing to move into code.

Prompt 2: check the tools the model can call

An example prompt for the tool handlers, once the threat model names them.

Task: review the tools that <the LLM feature> lets the model call:
<list the tool handlers>

For each tool, answer:
- Which arguments come from the model, and which from the server
  session or the record being worked on?
- Can an argument from the model reach another user's or tenant's data?
- Which identity does the tool use for the database or API, and what
  can that identity do beyond what the tool needs?
- Does it change data, move money or send anything? If so, where does
  a person approve it, and is that approval checked outside the model?
- Which fields does it return that the answer does not need?

For each finding, propose a fix and one test, for example a test that
calls the tool with another customer's ID and expects nothing back.
Do not edit files until I approve the fixes.
Never print, expose or commit secret values.
Do not commit, push, deploy, run destructive commands or make external
changes unless I explicitly authorise it.
If two reasonable attempts fail, stop and report the failure rather
than repeatedly guessing.

Read the findings against the code yourself before you trust the list.

Quick answers

What is threat modelling for LLM applications?

Threat modelling for LLM (large language model) applications is the usual four-question process applied to a feature that calls a model. You map the data flows, mark where untrusted text reaches the prompt, list what can go wrong, and agree a response for each threat.

Can prompt injection be fully prevented?

Prompt injection has no proven fool-proof prevention. The OWASP (Open Worldwide Application Security Project) LLM01:2025 entry says it is unclear whether one exists. Input filters, output checks, human approval and narrowly scoped tools reduce the risk and the damage, without removing either.

Should a system prompt be kept secret?

A system prompt should be treated as readable by users. The OWASP (Open Worldwide Application Security Project) LLM07:2025 entry says it should not be considered a secret or used as a security control. Keys and permission rules belong somewhere the model cannot reach.

Where should authorisation happen in an LLM feature?

Authorisation in an LLM (large language model) feature belongs in the tool handlers and the data layer. Server code checks the signed-in user’s identity and the record being worked on before it acts on what the model asked for.

How can LLM output leak data?

LLM (large language model) output can leak data when an app renders it as Markdown or HTML. An image whose URL carries private text sends that text to another server as soon as the browser loads it. Restrict image sources and show link addresses in full.

Is STRIDE enough for an LLM threat model?

STRIDE, a list of six threat categories, works as the prompt list for an LLM (large language model) threat model. The OWASP Top 10 for LLM Applications 2025 adds names such as excessive agency. OWASP’s Threat Modeling Cheat Sheet suggests the LINDDUN privacy method when privacy needs its own review.

More on AI coding is in the AI category.