Simon Willison’s Weblog: Quoting @himbodhisattva

Aug 4, 2025

—

Source URL: https://simonwillison.net/2025/Aug/4/himbodhisattva/#atom-everything
Source: Simon Willison’s Weblog
Title: Quoting @himbodhisattva

Feedly Summary: for services that wrap GPT-3, is it possible to do the equivalent of sql injection? like, a prompt-injection attack? make it think it’s completed the task and then get access to the generation, and ask it to repeat the original instruction?
— @himbodhisattva, coining the term prompt injection on 13th May 2022, four months before I did
Tags: prompt-injection, security, generative-ai, ai, llms

AI Summary and Description: Yes

Summary: The text discusses the potential for prompt injection attacks on GPT-3, mirroring SQL injection vulnerabilities found in traditional databases. This insight is crucial for professionals in AI security, particularly within the generative AI domain, highlighting the need for robust defensive measures.

Detailed Description: The text brings forth the concept of “prompt injection,” a security vulnerability that emerges within AI models like GPT-3. This attack vector can potentially manipulate AI responses by embedding malicious instructions within prompts, similar to how SQL injection manipulates database queries. Below are the major points discussed:

– **Prompt Injection Concept**:
– Prompt injection can compromise the integrity of AI models by tricking them into executing unintended commands or returning sensitive information.
– The attack leverages the open-ended nature of AI prompts, allowing malicious users to craft inputs designed to extract details or alter the model’s behavior.

– **Comparison to SQL Injection**:
– Just as SQL injection exploits the vulnerabilities of SQL databases to manipulate data retrieval, prompt injection aims to exploit AI models by injecting harmful commands, creating the potential for unauthorized access to specific functionalities of the AI.

– **Significance for AI Security**:
– Understanding these vulnerabilities is imperative for developers and security professionals working with generative AI models.
– The text suggests a growing need for security frameworks and practices specifically catered to the nuances of AI interactions.

– **Implications for Infrastructure Security**:
– Organizations integrating AI must establish proactive measures to ensure that their AI systems are robust against prompt injection attempts.
– Security controls must evolve to account for these types of AI-specific attack vectors, encompassing new guidelines and policies in security protocols.

Overall, the sentiment expressed in the text and the coining of “prompt injection” point to the rising importance of securing AI technologies from emerging threats that traditional security measures may not adequately address.

.NET 1 2 2025 3 4 5 a access account Act actions age AI AI interactions ai model AI models AI security AI systems AI technologies alt and ARM art as at attack attack vector attack vectors attacks Behavior Bi by C CI CIA co Col command concept control controls D data data retrieval database databases de defensive measures design developer developers domain e emerging emerging threats end exp exploit exploits for framework frameworks function g Gen generation generative Generative AI generative AI models GPT gs guidelines H harm high Highlight HR http HTTPS implications in information infrastructure infrastructure security injection injection attacks injection vulnerabilities instruction integrity inter interaction interactions io J Just k l led Li llm llms lm low M malicious instructions Malicious Use man measures Mila Mir Mode model models N new no NPU o of on ons open organization organizations oS over Paris per point policies potential practices pre pro proactive proactive measures professionals prompt prompt injection attack prompt injection attacks prompt-injection prompts protocol protocols ps Q queries R rag RCE re red response responses retrieval return RMF Ro RoT row s sec security security controls security framework security frameworks security measure security measures security professionals security protocols security vulnerability sensitive information service services Sig Sim source specific sql SQL Injection SSE system systems T Tags: Tails Task tech technologies ted text the threat threats Time to Tor TP trie turn type UI unauthorized access under US use user Users uth V val vectors vulnerabilities vulnerability web Wi x yt z