Google to Iran: Yes, we see you using Gemini for phishing and scripting. We’re onto you

Google says it’s spotted Chinese, Russian, Iranian, and North Korean government agents using its Gemini AI for nefarious purposes, with Tehran by far the most frequent naughty user out of the four.

The web giant has been tracking the use of Gemini by these nations, using not just simple things presumably like IP addresses to spot them but a combination of technical signals and behavioral patterns, we’re told.

And while these state-backed snoops have managed to use Gemini for translating and tailoring phishing lures for specific victims, looking up for information about surveillance targets, and writing some software scripts, Google admitted, the biz claims its guardrails at least stopped its AI from generating malware.

Overall, the American internet goliath reckons Iran et al aren’t doing anything too outrageous, and are mainly asking the LLM for info and guidance as it was designed for. In other words, foreign governments are using Google AI for bad things, but it’s not too bad, or so we’re told.

“While AI can be a useful tool for threat actors, it is not yet the gamechanger it is sometimes portrayed to be,” Google said in a Threat Intelligence Group (TIG) report [PDF] this week. “While we do see threat actors using generative AI to perform common tasks like troubleshooting, research, and content generation, we do not see indications of them developing novel capabilities.”

While AI can be a useful tool for threat actors, it is not yet the gamechanger it is sometimes portrayed to be

Iranian spies accounted for 75 percent of all observed Gemini use by the aforementioned quartet’s agents, the TIG report notes. The Google team identified over 10 Iran-backed cyber-crews using the AI service, with some particularly focused on researching Android-related security. More broadly, these groups used Gemini for reconnaissance, researching vulnerabilities, identifying free hosting providers, and crafting local personas and content for cyber operations. Notably, Iran’s APT42 unit leveraged Gemini to craft phishing content, making up 30 percent of all Iranian APT, or advanced threat actors, activity on the platform.

Chinese spies have also been using it for content creation and basic research, with 20 groups from the Middle Kingdom identified so far. Much of this activity focuses on researching US government institutions, while Beijing-backed snoops have also sought assistance with Microsoft-related systems and translation work, according to the report.

Google also says it has spotted North Korean operatives using its LLM to write job applications for IT workers as part of the hermit nation’s ongoing efforts to insert its workers into Western companies. Nine distinct groups of Norks also tried to find freelancer forums on Discord, and information related to South Korean military and nuclear technology, through Gemini.

Russians are relatively light users of Gemini, it seems, with only three groups observed by the team. Google speculates that this could be down to them either using domestically generated LLMs or attempting to limit exposure to avoid being monitored. Or maybe they’re just really good at hiding their usage of the LLM.

Around 40 percent of Russian activity came from operators linked to “Russian state-sponsored entities formerly controlled by the late Russian oligarch Yevgeny Prigozhin,” the cloud behemoth said. This presumably means the Wagner Group and its offshoots. Google notes a Russian operative used Gemini to generate and manipulate content, including rewriting articles with a pro-Kremlin slant for use in influence campaigns. This is exactly the sort of shenanigans Prigozhin’s Internet Research Agency used to do.

AI red-teaming tools helped X-Force break into a major tech manufacturer ‘in 8 hours’

Stifling Beijing in cyberspace is now British intelligence’s number-one mission

Putin’s pro-Trump trolls accuse Harris of poaching rhinos

Google brings more Gemini AI features to Android, saves the best for Pixel 9

When it comes to breaking Gemini’s guardrails and exploiting the engine to write malicious code or cough up personal information, Google claims the LLM is successfully blocking such attempts. It has noted an uptick in folks trying to use publicly known jailbreak prompts and then adapting them slightly in an attempt to get around the filters, but these appear ineffective.

The ad giant reported one case involved a request to embed encoded text in an executable and a separate attempt to generate Python code for a denial-of-service attack. While Gemini processed a Base64-to-hex conversion request, it refused further malicious queries.

Google has also detected attempts to use Gemini for researching methods to abuse its other services. The biz states its safety systems blocked these efforts, and that it is working on further improvements on these defenses. As well as this, its DeepMind wing is also mentioned in that the lab is apparently coming up with ways to protect AI services from attacks and prohibited queries.

“Google DeepMind also develops threat models for generative AI to identify potential vulnerabilities, and creates new evaluation and training techniques to address misuse caused by them,” the report added.

“In conjunction with this research, DeepMind has shared how they’re actively deploying defenses within AI systems along with measurement and monitoring tools, one of which is a robust evaluation framework used to automatically red team an AI system’s vulnerability to indirect prompt injection attacks.” ®