OpenAI Disrupts 5 AI-Powered, State-Backed Influence Ops

Most of the operations were feckless efforts with little impact, but they illustrate how AI is changing the game for inauthentic content on both the adversary and defense sides.

4 Min Read
A hand working puppet strings
Source: Spencer Whalen via Alamy Stock Photo

OpenAI has identified and upset five influence operations using its artificial intelligence (AI) tools in one way or another.

The various operations — from China, Iran, Israel, and two from Russia — focused on spreading political messaging. As OpenAI reports, they primarily used AI to generate text such as social media posts and comments

None of them were particularly effective, however. On the Brookings Breakout Scale, which measures the impact of influence operations on a scale of 1 to 6, none scored higher than a 2. A score of 1 means the campaign spread only within a single community or platform, while a 6 means triggering a policy response or some other form of concrete action, like violence. A 2 means the operation spread across multiple communities on one platform, or one community across multiple platforms.

The Current State of AI-Driven Influence Ops

The influence operations in question, while geographically diverse, ultimately were rather similar in nature:

  • Among the most notorious of them is Spamouflage, from China. It used OpenAI tooling to debug its code, research social media activity, and post content to X, Medium, and Blogspot in multiple languages.

  • Bad Grammar, a newly discovered threat from Russia, operated primarily on Telegram, targeting individuals in Eastern Europe and the United States. It also used AI to debug code it employed to run a Telegram bot and write political comments on Telegram in both Russian and English.

  • A second Russian group, Doppelganger, used AI to post comments on X and 9GAG in five European languages, plus generate headlines, and translate, edit, and convert news articles into Facebook posts.

  • An Iranian entity, known as the International Union of Virtual Media (IUVM), used AI for generating and translating articles, as well as headlines and website tags for its site.

  • Finally there's Zero Zeno, an operation run by Stoic, a Tel Aviv-based political marketing and business intelligence company. Stoic used OpenAI to generate articles and comments for Instagram, Facebook, X, and other websites.

Stoic has also drawn attention lately from Meta. In its latest "Adversarial Threat Report," Meta reported taking down 510 Facebook accounts, 32 Instagram accounts, 11 pages, and one group associated with the company. Only around 2,000 accounts followed its various Instagram accounts. About 500 accounts followed its Facebook pages, and less than 100 joined its Facebook group.

Overall, while useful as case studies, these campaigns won't be missed by many.

"Text-based campaigns are largely ineffective," says Jake Williams, former NSA hacker and faculty member at IANS Research, "because generative AI only helps scale those disinformation ops where people are reading content, something that's becoming increasingly rare, especially from sites without significant reputation. I think most people realize you can't trust everything you read on the Internet at this point. But images and video? That's what scares me. People are far more likely to consume synthetically created images and video than text.”

Why Tech Can't Stop Disinformation Ops

To combat greater AI misuse, OpenAI wrote in a more detailed report that it is collaborating with industry partners, and using threat activity to design more secure platforms for users. The company also "invest[s] in technology and teams to identify and disrupt actors like the ones we are discussing here, including leveraging AI tools to help combat abuses."

Dark Reading has reached out to OpenAI to clarify what it does, precisely, to disrupt and combat malicious actors, but has not yet received a reply.

Ultimately, the job of blocking fake content online is more technically difficult than many realize.

"One of the biggest challenges here is that reliable solutions [for] detection so far do not exist," explains Naushad UzZaman, CTO and co-founder of Blackbird.AI. "They are also unlikely to exist in the future. There have now been many cases of 'state-of-the-art' generated text detection applications being used to punish students. But many of these results are false. The problem is that the false positive rate of these detectors is too high. They flag too many instances of real text."

"The situation is also unlikely to improve in the future for two reasons," he continues. "First, any reliable detector of fake content can be used to create training data to improve the realism of the fake content generator. Second, the best way to make a fake content detector is actually to train a powerful fake content generator, and then use it for detection. By these two routes, any efforts to build better fake content detectors will lead to better fake content generators."

Williams agrees with the sentiment. "We cannot count on technical solutions here. Even digital watermarks are easily bypassed. This is a policy problem enabled by technology, where unfortunately technology won't be effective in addressing it," he says.

About the Author(s)

Nate Nelson, Contributing Writer

Nate Nelson is a freelance writer based in New York City. Formerly a reporter at Threatpost, he contributes to a number of cybersecurity blogs and podcasts. He writes "Malicious Life" -- an award-winning Top 20 tech podcast on Apple and Spotify -- and hosts every other episode, featuring interviews with leading voices in security. He also co-hosts "The Industrial Security Podcast," the most popular show in its field.

Keep up with the latest cybersecurity threats, newly discovered vulnerabilities, data breach information, and emerging trends. Delivered daily or weekly right to your email inbox.

You May Also Like

More Insights