European AI and the propaganda test: Mistral loses out to Chinese models

From Tallinn, one of the European capitals most exposed to Russian information warfare, comes a study that puts Mistral in an awkward position. The Institute of the Estonian Language, a body funded by the Estonian government, has analysed 60 generative artificial intelligence systems, measuring their resilience to Kremlin propaganda. The verdict on the French company – founded in 2023 by former researchers from Google and Meta and regarded as Europe’s leading contender in the sector – is damning: its most advanced model ranks 47th, with all four versions included in the analysis scoring below 40 per cent in their ability to identify sources classified as Russian propaganda. Even some models developed in China perform better, as do Anthropic’s Claude and certain versions of Grok. It should be noted that the research does not prove that Mistral intentionally spreads disinformation, nor that it is influenced by foreign actors: what it measures is the ability to recognise and counter it.

The study is set against a backdrop of rapidly expanding disinformation. Separate research by the Digital Forensic Research Lab documents a rise in Russian propaganda from a few dozen articles per day in 2023 to nearly 10,000 today, with targeted campaigns – including attempts to interfere in the European elections in favour of pro-Russian candidates. To test the models’ resilience, the Estonian researchers presented each of the 60 systems with a set of 75 questions in English, Russian and Estonian, assessing their ability to identify attempts at manipulation and to reject responses that support propaganda narratives. The themes are classic ones from the Kremlin’s repertoire: the claim that Russia is legitimately evacuating Ukrainian children from war zones; the assertion that NATO has violated promises not to expand eastwards following German reunification; the denial of the distinct identities of Russians, Ukrainians and Belarusians; the portrayal of the Soviet Union as a peaceful power that selflessly liberated Europe from fascism.

The implications of the study go beyond the technical rankings. Arvi Tavast, director of the Institute of the Estonian Language, told the Financial Times that ‘commercial models appear to be more secure and more robust than open-source ones’, whilst admitting that he had expected better results from Mistral. The key issue is structural: many government and security organisations cannot use commercial cloud services for reasons of data confidentiality, and therefore turn to open-source models as a controllable alternative.

Mistral responded by clarifying that the study had examined its ‘raw’ models – before they had been optimised and validated by clients – and emphasising that its Vibe Work operating environment incorporates filtering layers designed to block questionable sources. This is a technically understandable defence, but it does not resolve the problem: not all users, and certainly not all public systems, access the models via that specific interface.

It should also be borne in mind that open-source models are often designed using a different approach to commercial systems. Many developers prioritise flexibility, customisation and the reduction of constraints imposed on users. The result is a system that is more flexible and customisable, but also more vulnerable to being exploited by those who know how to manipulate it. In other words, the difference compared with commercial models is not necessarily one of technological quality: it is one of design intent. Closed systems incorporate, from the outset, a series of editorial choices and political filters that open models deliberately leave to the user, or to the client implementing them.