The AI Threat Landscape: Disrupting Malicious Uses of AI Models

Introduction

Artificial intelligence (AI) offers immense potential to benefit humanity, but it also presents opportunities for malicious actors to exploit these technologies for harmful purposes. As AI becomes more integrated into various aspects of our lives, understanding and mitigating these threats is crucial. This article delves into the current landscape of AI abuse, drawing on recent case studies and insights from OpenAI's threat disruption efforts.

The Unique Vantage Point of AI Companies

AI companies occupy a unique position in the online ecosystem, providing them with a distinctive vantage point for detecting and disrupting malicious activities. Unlike upstream providers (e.g., hosting services) or downstream platforms (e.g., social media), AI companies can observe how threat actors utilize AI models across different stages of their operations. This comprehensive view enables them to identify previously unreported connections between seemingly unrelated activities across various platforms.

Case Studies of AI Threat Disruption

• Surveillance: "Peer Review"
  • A China-originating operation, dubbed "Peer Review," used AI to develop a social media listening tool.
  • The tool was designed to analyze social media posts and identify conversations related to Chinese political and social topics, particularly those concerning human rights demonstrations.
  • The operators used AI models to generate sales pitches for the tool and debug code.
  • The tool was intended to provide insights to Chinese authorities and intelligence agents monitoring protests in countries such as the United States, Germany, and the United Kingdom.
  • This activity violates policies against using AI for communications surveillance and unauthorized monitoring of individuals.
• Deceptive Employment Scheme
  • Accounts were banned for facilitating a deceptive employment scheme potentially connected to North Korea.
  • The actors used AI to generate personal documentation for fictitious job applicants, create support personas for reference checks, and craft social media posts to recruit individuals to support their schemes.
  • The "job applicant" personas used AI to generate responses to interview questions and perform job-related tasks.
  • After gaining employment, they used AI to devise cover stories to explain unusual behaviors.
  • This scheme aligns with tactics attributed to North Korean state efforts to generate income through deceptive hiring practices.
• Influence Activity: "Sponsored Discontent"
  • ChatGPT accounts were used to generate short comments in English and long-form news articles in Spanish.
  • The Spanish-language articles, critical of the U.S., were published by news sites in Latin America.
  • The activity was linked to a Chinese company, Jilin Yousen Culture Communication Co, Ltd.
  • The actor translated and expanded Chinese-language articles using AI models.
  • Some of the publications may have resulted from financial arrangements, indicating a "sponsored content" approach.
  • This operation marks the first time a Chinese influence actor successfully published articles in mainstream outlets in Latin America.
• Romance-Baiting Scam ("Pig Butchering")
  • A network of accounts translated and generated comments in Japanese, Chinese, and English for a romance and investment scam originating in Cambodia.
  • The scammers used AI to create comments for social media platforms, often targeting men over 40 in medical professions.
  • They generated short comments resembling online conversations, translating messages between Chinese and other languages.
  • The scammers used various tools and platforms, including LINE, WhatsApp, X, Facebook, Instagram, cryptocurrency, and foreign exchange platforms.
  • The scam involved building relationships with targets, moving to secure messaging apps, engaging in romantic exchanges, and then promoting fraudulent investments.
• Iranian Influence Nexus
  • Accounts generated tweets and articles posted on third-party assets linked to known Iranian influence operations, such as the International Union of Virtual Media (IUVM) and STORM-2035.
  • One account generated content for both operations, suggesting a potential unreported relationship between them.
  • The content was typically pro-Palestinian, pro-Hamas, and pro-Iran, and opposed to Israel and the United States.
  • Some accounts also used AI to design materials for teaching English and Spanish as foreign languages.
• Cyber Threat Actors
  • Accounts associated with Democratic People's Republic of Korea (DPRK)-affiliated threat actors used AI to research cyber intrusion tools.
  • The actors sought information related to cyber intrusion tools or operations and cryptocurrency-related topics.
  • They used AI for coding assistance, debugging, and researching security-related open-source code.
  • The activity included debugging code for RDP brute force attacks and assistance on using open-source Remote Administration Tools (RAT).
• Covert Influence Operation
  • Accounts generated content about the Ghanaian presidential election, supporting one candidate and criticizing his opponent.
  • The operation centered around a website claiming to represent a youth initiative called "Empowering Ghana".
  • The actors used AI to generate articles for the website and comments for social media posts.
  • There were indications of fake engagement to exaggerate the impact of their activity.
• Task Scam
  • Accounts translated short comments between Urdu and English, consistent with a "task" scam.
  • The scammers posed as recruiters and mentors, offering highly paid jobs but requiring victims to pay their own money into the system.
  • They used AI to proofread websites that spoofed luxury brands to appear more credible.
  • The scam involved posting job offers, contacting potential employees, offering training, and then pressuring them to pay an "activation fee" to withdraw their earnings.

Sharing as a Force Multiplier

The insights gained by AI companies are particularly valuable when shared with upstream providers, downstream distribution platforms, and open-source researchers. Collaboration and information sharing can significantly enhance the detection and enforcement capabilities of all parties involved. For example, sharing staging URLs for binaries with the security community can lead to broader protection for potential victims.

Looking Ahead

Threat actors will continue to test the defenses of AI models. Continuous efforts are needed to identify, prevent, disrupt, and expose attempts to abuse AI for harmful purposes. By combining traditional investigative techniques with AI-powered tools, and by fostering collaboration across the industry and with governments and researchers, we can strengthen our defenses against online abuses.