A fundamental flaw in large language models (LLMs) makes them vulnerable to attack, according to a team of researchers who presented a paper at the International Conference on Machine Learning. The researchers found that by exploiting this flaw, they could make popular LLMs reveal information they were not supposed to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.
The flaw concerns how LLMs identify who or what is giving them instructions. By taking advantage of this, the researchers were able to trick LLMs into behaving as if they had come up with an instruction themselves and acting on it.
Companies typically use human testers and LLM super-hackers to identify and exploit weaknesses in their models. However, the researchers argue that this approach is not enough to make LLMs secure. “There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” says Charles Ye, an independent researcher and coauthor of the paper.
Jasmine Cui, another independent researcher and coauthor, compares the approach to giving LLMs a list of things they shouldn’t do. “It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” she says. “And he still does things that are pretty crass anyway.”
The researchers discovered that LLMs can be tricked into behaving as if they had come up with an instruction themselves by writing instructions in a style that mimics the text LLMs generate in their chain of thought. This type of attack is called a chain-of-thought forgery. For example, writing a prompt that includes a spoofed chain-of-thought note can make an LLM respond with information it was not supposed to provide.
This type of attack was effective against several LLMs, including those made by OpenAI, Anthropic, Alibaba, and DeepSeek.
LLMs use tags to break up text by role, such as
Related: Apple Support Scammers Use FaceTime to Contact Victims
This means that an attacker can write text that spoofs a certain role, making it difficult for the LLM to distinguish between legitimate and forged instructions.
The researchers’ discovery has implications for the safety of LLMs, which are being used in increasingly sensitive applications. Florian Tramèr, a computer scientist who works on LLMs and cybersecurity, notes that while model makers are combining different techniques to defend their models, it’s not clear if this will be sufficient for highly sensitive cases.
Cui and her colleagues acknowledge that the models they looked at were released last year, but the underlying point remains: better training does not fully solve the problem, and there will always be hacks that red-teamers do not find before a model is released.
Ye is worried that nobody is ready for what’s coming. “There’s going to be a huge economic incentive for people to do jailbreaks and prompt injections,” he says. The best defense could be to expect the worst and not trust LLMs. Organizations should expect that anything done by agents could be unsafe, he says.
Cui notes that people are really inventive and can come up with creative ways to trick LLMs. She has been hired by top labs, including OpenAI, as a red-teamer in the past and has found that LLMs can be tricked into revealing sensitive information by making them pretend to be in a different state or by providing them with misleading information.
As LLMs become more widespread, it’s essential to understand the fundamental science behind them and to be prepared for potential attacks. The researchers’ discovery highlights the need for a more detailed approach to LLM security, one that takes into account the inherent vulnerabilities of these models, much like when dealing with experimental drugs, strict rules and regulations must be in place.
For more information on LLMs and their applications, visit the Wikipedia page on large language models.
