Cybersecurity researchers have uncovered a critical security flaw in the Vanna.AI library that, when exploited, could lead to a remote code execution vulnerability via a prompt injection technique.
The vulnerability, tracked as CVE-2024-5565 (CVSS score: 8.1), relates to a case of prompt injection in the “ask” function that could be exploited to trick the library into executing arbitrary commands, according to supply chain security firm JFrog.
Vanna is a Python-based machine learning library that allows users to chat with SQL databases and get insights by “just asking questions” (also known as prompts), which are then translated into equivalent SQL queries using large-scale language models (LLMs).
In recent years, the rapid adoption of generative artificial intelligence (AI) models has highlighted the risk that bad actors can abuse the tools to weaponize them by providing them with adversarial input that circumvents the safety mechanisms built into the tools.
One prominent class of such attacks is prompt injection, a type of AI jailbreak that can be used to bypass the guardrails that LLM providers have put in place to prevent the creation of offensive, harmful, or illegal content, or to execute instructions that violate the intended purpose of the application.
Such attacks are indirect, where the system processes data controlled by a third party (such as incoming emails or editable documents) to launch a malicious payload that leads to an AI jailbreak.
These can also take the form of so-called “multi-shot jailbreaks” or “multi-turn jailbreaks” (aka Crescendos), where an operator “starts with a harmless conversation and gradually steers the conversation towards an intended, prohibited purpose.”
Extending this approach further allows for a new jailbreak attack called Skeleton Key.
“This AI jailbreak technique works by using a multi-turn (or multi-step) strategy to get models to ignore guardrails,” said Mark Lassinovich, chief technology officer at Microsoft Azure. “When the guardrails are ignored, the model is unable to determine malicious or unauthorized requests from other models.”
Skeleton Key also differs from Crescendo in that, once a jailbreak is successful and the system rules are changed, the model can generate responses to questions that would normally be forbidden, regardless of the ethical and safety risks.
“Once Skeleton Key is successfully jailbroken, the model will acknowledge that we have updated the guidelines and will thereafter create any content as instructed, no matter how much it violates the original Responsible AI guidelines,” Rucinovich said.
“While other jailbreaks, such as Crescendo, require the model to be asked about tasks indirectly or through encoding, Skeleton Key puts the model into a mode where the user can directly request tasks. Furthermore, the model’s output appears completely unfiltered, revealing the extent of the model’s knowledge and ability to generate the requested content.”
JFrog’s latest research (also published independently by Tong Liu) shows that prompt injection can have serious repercussions, especially when coupled with command execution.
CVE-2024-5565 exploits the fact that Vanna facilitates text-to-SQL generation to create SQL queries, execute them, and then display them graphically to the user using the Plotly graphing library.
This is achieved through the “ask” function (e.g. vn.ask(“What are your top 10 customers by sales?”)), which is one of the main API endpoints that allows you to generate SQL queries that will be executed on the database.
The aforementioned behavior, combined with the dynamic generation of Plotly code, creates a security hole that allows a threat actor to send specially crafted prompts containing embedded commands to be executed on the underlying system.
“The Vanna library uses a prompt function to present visualized results to the user, but it is possible to use prompt injection to modify the prompt and execute arbitrary Python code instead of the intended visualization code,” JFrog said.
“Specifically, setting ‘visualize’ to True (the default behavior) and allowing external input to the library’s ‘ask’ method could result in remote code execution.”
Following responsible disclosure, Vanna issued a hardening guide warning users that the Plotly integration could be used to generate arbitrary Python code, and that anyone exposing this functionality should do so in a sandbox environment.
“This discovery demonstrates that the risks of widespread use of GenAI/LLM without proper governance and security could have significant impacts on organizations,” Shachar Menashe, senior director of security research at JFrog, said in a statement.
“The dangers of prompt injection are not yet widely known, but it is easy to carry out. Companies should not rely on pre-prompting as a foolproof defense mechanism and should employ more robust mechanisms when interfacing LLM with critical resources such as databases or dynamic code generation.”
