Simon Willison’s Weblog: The Summer of Johann: prompt injections as far as the eye can see

Common patterns
There are a number of patterns that show up time and time again in the above list of disclosures:

Prompt injection. Every single one of these attacks starts with exposing an LLM system to untrusted content. There are so many ways malicious instructions can get into an LLM system – you might send the system to consult a web page or GitHub issue, or paste in a bug report, or feed it automated messages from Slack or Discord. If you can avoid unstrusted instructions entirely you don’t need to worry about this… but I don’t think that’s at all realistic given the way people like to use LLM-powered tools.

Exfiltration attacks. As seen in the lethal trifecta, if a model has access to both secret information and exposure to untrusted content you have to be very confident there’s no way for those secrets to be stolen and passed off to an attacker. There are so many ways this can happen:

The classic Markdown image attack, as seen in dozens of previous systems.
Any tool that can make a web request – a browser tool, or a Bash terminal that can use curl, or a custom view_text_website tool, or anything that can trigger a DNS resolution.
Systems that allow-list specific domains need to be very careful about things like *.azure.net which could allow an attacker to host their own logging endpoint on an allow-listed site.

Arbitrary command execution – a key feature of most coding agents – is obviously a huge problem the moment a prompt injection attack can be used to trigger those tools.

Privilege escalation – several of these exploits involved an allow-listed file write operation being used to modify the settings of the coding agent to add further, more dangerous tools to the allow-listed set.

The AI Kill Chain
Inspired by my description of the lethal trifecta, Johann has coined the term AI Kill Chain to describe a particularly harmful pattern:

prompt injection leading to a

confused deputy that then enables
automatic tool invocation

The automatic piece here is really important: many LLM systems such as Claude Code attempt to prevent against prompt injection attacks by asking humans to confirm every tool action triggered by the LLM… but there are a number of ways this might be subverted, most notably the above attacks that rewrite the agent’s configuration to allow-list future invocations of dangerous tools.
A lot of these vulnerabilities have not been fixed
Each of Johann’s posts includes notes about his responsible disclosure process for the underlying issues. Some of them were fixed, but in an alarming number of cases the problem was reported to the vendor who did not fix it given a 90 or 120 day period.
Johann includes versions of this text in several of the above posts:

To follow industry best-practices for responsible disclosure this vulnerability is now shared publicly to ensure users can take steps to protect themselves and make informed risk decisions.

It looks to me like the ones that were not addressed were mostly cases where the utility of the tool would be quite dramatically impacted by shutting down the described vulnerabilites. Some of these systems are simply insecure as designed.
Back in September 2022 I wrote the following:

The important thing is to take the existence of this class of attack into account when designing these systems. There may be systems that should not be built at all until we have a robust solution.

It looks like we built them anyway!
Tags: security, ai, prompt-injection, generative-ai, llms, exfiltration-attacks, johann-rehberger, coding-agents, lethal-trifecta

AI Summary and Description: Yes

Summary: The text highlights a series of vulnerabilities related to various AI tools and demonstrates the ongoing risks associated with prompt injection attacks in machine learning systems. It showcases the proactive efforts of researcher Johann Rehberger, emphasizing the critical need for heightened security measures in AI tools used across various applications. This is especially relevant for professionals focusing on AI security, infrastructure, and compliance.

Detailed Description:
This analysis responds to the prominent vulnerabilities identified by Johann Rehberger under the theme “The Month of AI Bugs.” His research illustrates the pervasive threat posed by prompt injection attacks, which exploit various AI tools to access sensitive data and execute arbitrary commands. The following points summarize the main topics of discussion:

– **Vulnerability Overview**:
– The report encompasses vulnerabilities across several AI systems, including ChatGPT, Codex, and others.
– Each entry discusses specific problems and the consequences of these vulnerabilities, ranging from data exfiltration to arbitrary command executions.

– **Common Attack Patterns Identified**:
– **Prompt Injection**:
– All attacks exploit untrusted content being processed by LLM systems, leading to unauthorized actions.
– This method potentially allows malicious inputs from various sources, including web pages and chat platforms.

– **Data Exfiltration Techniques**:
– Noteworthy methods include Markdown image exploits and tools capable of making web requests.
– The risk is heightened when systems have access to sensitive data while being exposed to untrusted content.

– **Arbitrary Command Execution**:
– Many coding agents are vulnerable to attacks that could trigger command executions due to flawed safeguards.

– **Privilege Escalation**:
– Some vulnerabilities allow for escalation of privileges, where attackers can bypass restrictions to gain greater control.

– **Formation of the “AI Kill Chain”**:
– Rehberger introduces the “AI Kill Chain” concept, illustrating how prompt injection can lead to initiating harmful automatic tool actions.
– Recognizing the connection between injection attacks and system autonomy highlights the dual risks of exploitation and misuse.

– **Industry Response**:
– The text highlights challenges in responsible vulnerability disclosure; while some issues were addressed, many remain unresolved.
– There is a critique regarding the security measures incorporated within the design of certain AI systems; the lack of robust security solutions could render these tools inherently insecure.

– **Practical Implications**:
– Organizations and professionals involved in AI development and deployment must recognize the evolving threat landscape, particularly in regard to prompt injection vulnerabilities.
– It underscores the necessity for stringent security protocols and thoughtful design considerations to prevent similar vulnerabilities in the future.

This comprehensive examination not only sheds light on current vulnerabilities in AI systems but also serves as a cautionary tale for security stakeholders to enhance compliance and governance practices surrounding AI technologies.