How do hacker attacks on corporate artificial intelligence systems work?

A few hours after LLMs such as chatGPT hit the headlines, experts began to report that they would be used by cyber criminals to carry out their misdeeds. The alarm, however, was mainly about the fact that they would be used to create more credible fake e-mails, forge documents in different languages, and generally act as a support to make criminals more effective in carrying out the kind of attacks that were already plaguing us. But every new technology brings with it vulnerabilities, and now the time has come for criminals to use artificial intelligence systems used in the company to steal documents or attack the internal network.

The first case came in June, with a vulnerability called EchoLeak affecting Microsoft CoPilot 365. Specifically, EchoLeak exploited the way Copilot for Microsoft 365 automatically fetches the 'context' for requests made to it from e-mails and documents. An error in the handling of content read from e-mails allowed an attacker to send a seemingly harmless message that contained hidden instructions for the AI. The user did not have to do anything in particular except to continue using his computer normally: when, even days later, Copilot was asked a question that related to a topic dealt with in the e-mail sent by the attacker, the system would retrieve it and start following the hidden instructions, collecting internal data (from Outlook/SharePoint/Teams) and then sending it externally, for instance via links or images that generate automatic requests to a server controlled by the attacker. It was in effect a so-called 'zero-click' vulnerability because it allows an attack without the user consciously opening or clicking something. Microsoft corrected the problem, but the Pandora's box was uncovered: there are systems in companies that need to be protected differently from others because they are subject to attacks of a very different kind.

A few days ago, a similar problem to EchoLeak was also found in Gemini. Again, no user interaction was needed, and the attack was carried out via a simple e-mail invitation for Google Calendar. An invitation 'packaged' in the right way, in fact, could contain instructions that would be executed by Gemini when asked about events such as 'what appointments do I have on my calendar for today'? The instructions that it was possible to give to Gemini had no limits and this meant that one could also control home automation devices possibly connected to the affected account. Again, research seems to have got there before the criminals.

Another 'unconventional' attack system was discovered on an Asana AI system and revealed a few days after Echoleak's discovery. Basically, Asana had activated an 'experimental' AI integration based on MCP (Model Context Protocol) to allow models with company data to talk to other apps. Companies would upload their data and employees or customers could ask questions to be answered based on the uploaded documents. Unfortunately, there was a data isolation bug in that MCP server: under certain conditions, information from your Asana domain could end up visible to other users using the same AI integration but belonging to different companies. For this reason, Asana shut down the service from 5 to 17 June, reset all connections and warned customers that they could safely start using the service again after a few days. While in the case of Copilot, the problem was in the way the instructions were fished, in this case the AI system could draw on the wrong documents because the boundaries separating the information of one company from that of another had not been well established. A problem that could have led to leaks of confidential information, but which seems to have been identified before it could do any damage.

What we have seen so far are attacks of a different kind, called 'LLM Scope Violation', where instructions are used to make the AI do things it should not do. In the case of LameHug, however, we are faced with what appears to be the first 'AI malware' seen so far. Indeed, it must arrive on corporate systems via traditional attack vectors, but once inside, it exploits one of AliBaba's AI systems that specialises in writing executable code to try to accomplish its mission of compromising systems. Through the commands it executes, it collects basic information about the system where it is running (such as active processes, hardware and network connection information), recursively searches for Microsoft Office documents in the 'Desktop', 'Documents' and 'Downloads' folders, and exfiltrates the collected data via Ftp or Post. Basically, it tries to do 'alone' what a criminal does when he manages to breach a company's computer defences.