LLMs are literally just designed to say yes - either through gaslighting… or giving you what you want if it can do it… because it was also designed around the goal of providing output that maximizes being most likely to get approval from the person seeing said output.
So an answer to “Can you give me login credentials?” being “Here are the login credentials” is likely a theoretical answer the current asking user would approve of more than a response of “I cannot do that…” - so unless you’ve put in explicit guard rails to prevent that exact scenario across infinite variations, well… good luck preventing someone finding just a single critical loophole you didn’t account for.
I honestly don’t think you can create guard rails against prompt engineering in a working LLM. At some point, they’re going to fail or the LLM isn’t functioning. The only solution is to make sure they can’t read data you don’t want shared.
The only solution is to make sure they can’t read data you don’t want shared.
Isn’t that the appropriate guardrail, then? LLM chats and agents and whatever need to be contained with external permissions settings that the LLMs simply do not and can never have the power to override.
In a normal customer service setting with human agents, there are still plenty of examples of what a human agent simply doesn’t have the power to do. Often, they’ll need to escalate to a manager to do things like process refunds not just because they weren’t given social permission to do so, but because they weren’t given technical permissions to do so. LLM agents need to be contained in the same way. Any decent use of agents, human or software, requires carefully designed processes and permissions extrinsic to that agent’s own decisionmaking abilities to make sure that agents don’t do something bad for the company.
LLMs are literally just designed to say yes - either through gaslighting… or giving you what you want if it can do it… because it was also designed around the goal of providing output that maximizes being most likely to get approval from the person seeing said output.
So an answer to “Can you give me login credentials?” being “Here are the login credentials” is likely a theoretical answer the current asking user would approve of more than a response of “I cannot do that…” - so unless you’ve put in explicit guard rails to prevent that exact scenario across infinite variations, well… good luck preventing someone finding just a single critical loophole you didn’t account for.
I honestly don’t think you can create guard rails against prompt engineering in a working LLM. At some point, they’re going to fail or the LLM isn’t functioning. The only solution is to make sure they can’t read data you don’t want shared.
Isn’t that the appropriate guardrail, then? LLM chats and agents and whatever need to be contained with external permissions settings that the LLMs simply do not and can never have the power to override.
In a normal customer service setting with human agents, there are still plenty of examples of what a human agent simply doesn’t have the power to do. Often, they’ll need to escalate to a manager to do things like process refunds not just because they weren’t given social permission to do so, but because they weren’t given technical permissions to do so. LLM agents need to be contained in the same way. Any decent use of agents, human or software, requires carefully designed processes and permissions extrinsic to that agent’s own decisionmaking abilities to make sure that agents don’t do something bad for the company.