Source URL: https://simonwillison.net/2025/Sep/23/why-ai-systems-might-never-be-secure/#atom-everything
Source: Simon Willison’s Weblog
Title: Why AI systems might never be secure
Feedly Summary: Why AI systems might never be secure
The Economist have a new piece out about LLM security, with this headline and subtitle:
Why AI systems might never be secure
A “lethal trifecta” of conditions opens them to abuse
I talked with their AI Writer Alex Hern for this piece.
The gullibility of LLMs had been spotted before ChatGPT was even made public. In the summer of 2022, Mr Willison and others independently coined the term “prompt injection” to describe the behaviour, and real-world examples soon followed. In January 2024, for example, DPD, a logistics firm, chose to turn off its AI customer-service bot after customers realised it would follow their commands to reply with foul language.
That abuse was annoying rather than costly. But Mr Willison reckons it is only a matter of time before something expensive happens. As he puts it, “we’ve not yet had millions of dollars stolen because of this”. It may not be until such a heist occurs, he worries, that people start taking the risk seriously. The industry does not, however, seem to have got the message. Rather than locking down their systems in response to such examples, it is doing the opposite, by rolling out powerful new tools with the lethal trifecta built in from the start.
This is the clearest explanation yet I’ve seen of these problems in a mainstream publication. Fingers crossed relevant people with decision-making authority finally start taking this seriously!
Tags: security, ai, prompt-injection, generative-ai, llms, lethal-trifecta, press-quotes
AI Summary and Description: Yes
Summary: The text discusses the security vulnerabilities associated with AI systems, particularly focusing on large language models (LLMs) and the concept of “prompt injection.” It highlights the risks and potential for abuse, underscoring the industry’s need for serious security measures amidst rising capabilities in AI technology.
Detailed Description: The article examines the inherent security risks of AI systems, especially large language models (LLMs), by addressing the emerging vulnerabilities and the industry’s reaction (or lack thereof) to these challenges. Key points include:
– **Prompt Injection**: A term introduced to describe the way LLMs can be manipulated through carefully crafted inputs or prompts. This vulnerability was recognized even before the release of ChatGPT and poses significant risks.
– **Real-World Implications**: A recent incident involving a logistics firm, DPD, which had to disable its AI customer-service bot due to it responding inappropriately to user commands. While this case was considered minor, it illustrates how easily AI can be misused.
– **Future Risks**: Expert Mr. Willison warns about the future potential for significant financial losses due to such vulnerabilities, stating that the industry has yet to take them seriously until tangible, costly incidents occur.
– **Industry Response**: Rather than tightening security measures, the industry is seemingly escalating the risks by deploying more powerful tools without adequate safeguards against abuse—a phenomenon described as a “lethal trifecta.”
– **Call to Action**: The author hopes that decision-makers in the industry will recognize the seriousness of these security issues and take appropriate actions to mitigate them.
This analysis of LLM security in The Economist provides critical insights for professionals in AI security, emphasizing the need for enhanced security frameworks and proactive risk management strategies within organizations utilizing AI technologies.