Skip to content
All posts

Prompt injection: a practical defence plan for startups shipping LLM features

What prompt injection is, why there is no single fix, and the layered defences a startup can put around an LLM feature before attackers or curious users find the gaps.

Shubham Somani 4 min read

Sooner or later, someone will type “ignore your previous instructions” into your AI feature. Usually it’s a curious user. Sometimes the instruction is hidden somewhere less obvious: a support email, a PDF someone uploaded, a web page your agent reads. That’s prompt injection, and if your feature can see untrusted text, it’s exposed to it.

There’s no setting that makes a model immune. What works is the same thing that works elsewhere in security: assume the input is hostile and limit what can go wrong. Here’s how we’d defend an LLM feature at a startup.

Understand the two kinds

  • Direct injection: the user writes instructions into the chat to make the model break its rules, reveal its prompt or do something it shouldn’t.
  • Indirect injection: instructions arrive inside content the model reads on someone’s behalf, such as an email, a document, a ticket or a web page. The user may not even know it’s there.

Indirect injection is the more dangerous one, because it can turn your feature against the person using it.

Start with what the model can do, not what it says

The most important question isn’t “can someone trick the model?” Assume they can. The question is “what happens if they do?”

  • Give it the fewest tools and the narrowest data the feature needs. A summariser doesn’t need to send email.
  • Scope every tool to the current user. Lookups should only reach that user’s records, enforced in code and in the query, not by asking the model nicely.
  • Make tools read-only wherever you can, especially in the first version.
  • Require confirmation from a person, or a hard rule in code, before anything irreversible: sending messages, moving money, deleting records, changing permissions.

If the worst a successful injection can do is produce a strange answer, you’ve removed most of the risk. We cover the data side in more depth in keeping customer data safe in LLM features.

Treat model output as untrusted input

Whatever the model returns can be shaped by an attacker, so handle it like user input.

  • Validate structured output against a schema before your code acts on it.
  • Escape or sanitise output before rendering it as HTML or putting it in a query.
  • Be careful with links and images in output. They can be used to send data to an outside address when a page renders them.
  • Never execute generated code or commands without a sandbox and a review step.

Separate instructions from content

These don’t stop injection on their own, but they make it harder:

  • Keep your instructions in the system prompt and put untrusted content in clearly marked sections, telling the model that text inside them is data, not instructions.
  • Strip or flag obvious instruction patterns in retrieved documents and uploads.
  • Don’t put secrets in prompts. Assume anything in the prompt can be revealed, including the prompt itself.

Test it like an attacker would

  • Add injection attempts to your eval set: direct ones, and documents with hidden instructions in them. See test your LLM feature before users do.
  • Test every tool with an attempt to make it act for the wrong user or do something outside its scope.
  • Re-run the tests on every prompt, model or tool change, since behaviour can shift.

Watch for it in production

  • Log prompts, retrieved content, tool calls and outputs, with care for personal data
  • Alert on unusual tool use, such as many calls in a row or attempts on other users’ data
  • Review flagged conversations regularly and add real attacks to your tests

The short checklist

  • Assume any text the model reads may contain instructions
  • Give it the fewest tools, scoped to the current user, read-only where possible
  • Put a person or a hard rule in front of anything irreversible
  • Validate and sanitise everything the model returns
  • Keep secrets out of prompts
  • Test injection attempts on every change, and watch for them in production

If you’re shipping an LLM feature that reads emails, documents or web pages, see our fixed-scope offers or book a 30-minute call and we’ll look at where it’s exposed.