Prompt injection – Everyone should know this about LLM systems.

With Artificial Intelligence (AI) and Machine Learning (ML) advancements in recent years, algorithms have gained unparalleled power to shape our digital world; thus, AI hacking has gone beyond science fiction and become an actual reality of our daily lives. There is a positive correlation between AI’s capabilities and the risk of exploitation. The more developed AI capacities, the higher the risk of exploitation. In this era of flourishing technological advancements, protecting the integrity and reliability of AI applications is of utmost importance. Prompt injection is one of many potential vulnerabilities that could pose a significant threat to humanity. And, while it is often disregarded, a quick injection can have catastrophic consequences if ignored.

What is a prompt?

To better understand prompt injection, we need to understand what a prompt is. A prompt is a series of instructions given to an AI language model on how the model should go about creating a response. They serve as a starting point for dialogue between a human and an AI model, determining the direction of the model’s outputs. The precision and relevance of the responses are particularly influenced by the instructions provided by the prompt.

What is a prompt injection?

Prompt injection is a cybersecurity challenge affecting AI/ML models, particularly those using prompt-based learning. OWASP defines a prompt injection attack as “using carefully crafted prompts that make the model ignore previous instructions or perform unintended actions.” It occurs when outsiders manipulate inputs provided by the prompts to the model, causing the model to violate its original commands or perform activities it was not supposed to do. This tweak exploits how these models comprehend and perform tasks by adhering to written guidelines (prompts). Attackers can deceive and manipulate the model into taking undesirable actions or revealing sensitive information by feeding harmful signals to the AI system.

Types of prompt injections

Prompt injections can be done in two ways: Indirect or Direct. The act of inserting malicious prompts or text into a public source, which an LLM (Large Language Model) may later access or read, is known as an Indirect prompt injection. This strategy is more undercover since it waits for the malicious input to naturally come into contact with the target system rather than interfering with it immediately. For instance, inserting malicious prompts into text on a webpage that an LLM would use to gather information could be considered an Indirect injection.

On the other side, Direct Prompt Injections include giving malicious prompts to the target system or model in order to interact with it directly. This might be as simple as giving an AI chatbot misleading instructions or inquiries. Direct injections try to have an effect right away by actively interacting with the model to change its behavior or output.

Prompt injection and AI biases

Prompt injection can be used to reduce biases in AI models or to sway user involvement in one way or another. To manipulate AI responses, businesses might provide prompts that direct the system to match marketing plans or company goals. Decision-making processes may be impacted by this, since it may change user opinions or judgments based on AI-generated content.

On the other side, businesses can purposefully utilize prompt injection to present a greater diversity of viewpoints or more neutral language in order to counteract AI biases. They seek to lessen the possibility of spreading incorrect or misleading information and encourage more accurate, objective outputs by modifying the prompts provided to AI models.

Is it ethical, though?

AI bias occurs when discriminatory data and algorithms are embedded within AI models, leading to biased decisions and actions that reflect these prejudices.

As previously stated, prompt injection entails decorating the input prompts in order to produce particular replies. Among other undesirable outcomes, this manipulation could lead to the creation of offensive content, the spread of misinformation, or the emergence of security vulnerabilities. An example of this is when AI systems used in hiring for law enforcement inadvertently reinforce racial or gender biases due to biased training data. Another example could be the ability to force an AI model into a “jailbreak” state, in which it acts beyond the ethical and safety limitations that its creators intended and bypasses its moderation framework. These flaws could be used to carry out illegal transactions or cause the AI to produce inaccurate data.

However, prompt injection can be advantageous as well. By carefully planning prompt injections, our capacity to guide massive language models to generate accurate and relevant content boosts their utility to users. Imagine asking a Gen AI to produce an image of a firefighter. A “non-prompt injected system” will probably show images of white, well-trained men wrestling with fire, while a “prompt injected system” will perhaps display a more balanced result showing both men and women of different races. Most of us will likely be happy to get images of a diverse set of firefighters. The problem, though, is that the system does not know if you are searching for historically accurate information or a utopian future that we aim to create.

Conclusion

Exploring AI presents us with amazing benefits, but it also highlights some risks, including prompt injection. This challenge draws attention to the intricate technical problems that arise when developing AI and emphasizes how crucial it is to use AI ethically. Users must be made aware of these risks in order to be ready and able to respond appropriately. On the other hand, there may be advantages to the prompt injection phenomenon. This is just the beginning of the conversation. Working together with professionals, programmers, and academics, we can make good use of quick injection. This cooperative strategy seeks to leverage AI developments for the benefit of both users and society as a whole.

My question, though, remains: is it ethical to inject prompts even though the decision has been strategically made in order to mitigate the effects of the existing biased data?

/Nashmil Mobasseri – Digital Product Designer and Accessibility advocate with a background in software development @ Softhouse

…