Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


End of Bibblio RCM includes -->
02:45 PM
Connect Directly

When AI Becomes the Hacker

Bruce Schneier explores the potential dangers of artificial intelligence (AI) systems gone rogue in society.

Source: Bruce Schneier
Source: Bruce Schneier

For the past couple of years, renowned technologist and researcher Bruce Schneier has been researching how societal systems can be hacked, specifically the rules of financial markets, laws, and the tax code. That led him to his latest examination of the potential unintended consequences of artificial intelligence on society: how AI systems themselves, which he refers to as "AIs," could evolve such that they automatically - and inadvertently - actually abuse societal systems.

"It's AIs as the hacker," he says, rather than hackers hacking AI systems.

Schneier will discuss his AI hacker research in a keynote address on Monday at the 2021 RSA Conference, which, due to the pandemic, is being held online rather than in person in San Francisco. The AI topic is based on a recent essay he wrote for the Cyber Project and Council for the Responsible Use of AI at the Belfer Center for Science and International Affairs at Harvard Kennedy School.

The core question Schneier asks is this: What if artificial intelligence systems could hack social, economic, and political systems at the computer scale, speed, and range such that humans couldn't detect it in time and suffered the consequences?

It's where AIs evolve into "the creative process of finding hacks."

"They're already doing that in software, finding vulnerabilities in computer code. They're not that good at it, but eventually they will get better [while] humans stay the same" in their vulnerability discovery capabilities, he says. 

In less than a decade from now, Schneier predicts, AIs will be able to "beat" humans in capture-the-flag hacking contests, pointing to the DEFCON contest in 2016 when an AI-only team called Mayhem came in dead last against all-human teams. That's because AI technology will evolve and surpass human capability.

Schneier says it's not so much AIs "breaking into" systems, but AIs creating their own solutions. "AI comes up with a hack and a vulnerability, and then humans look at it and say, 'That's good,'" and use it as a way to make money, like with hedge funds in the financial sector, he says.

The irony here, of course, is that AI starts with human input and programming. Frankenstein analogies aside, the core problem is that AI doesn't have the same human cognitive functions like empathy or a gut check to know where to draw the line. Schneier notes that while there's plenty of research on incorporating context, ethics, and values into AI programs, it's not a built-in function of today's AI systems.

Even so, he says, humans will employ AI to find loopholes in tax codes, such as a major accounting firm doing so to find a new "tax dodge to sell to their customers." So financial firms aren't likely to "program in" rules that thwart their ability to monetize AI knowledge.

The biggest risk is that AIs will find a way around a rule without humans knowing - "that AIs will figure out something that will hack the rules and we won't realize that," Schneier says.

Schneier points to the Volkswagen scandal in 2015, when the carmaker was caught cheating on emissions control-level tests of its vehicle models after engineers programmed the cars' computer systems to activate emissions-curbing only during tests, not in its normal operations. 

"There it was humans attacking the rules" and not AI itself, he says, but it's a good example of what AI could ultimately do to cheat a system if left unchecked to learn ways around it.

In his essay, "The Coming AI Hackers," Schneier describes it this way: "If I asked you to design a car's engine control software to maximize performance while still passing emissions control tests, you wouldn't design the software to cheat without understanding that you were cheating. This simply isn't true for an AI; it doesn't understand the abstract concept of cheating. It will think 'out of the box' simply because it won't have a conception of the box, or of the limitations of existing human solutions. Or of ethics. It won't understand that the Volkswagen solution harms others, that it undermines the intent of the emissions control tests, or that it is breaking the law."

A Wake-up Call to Action
Schneier admits the concept of AIs as hackers is "super speculative" for now, but it's an issue that needs to be addressed. 

"We need to think about this," he says. "And I'm not sure you can stop this. The ease of this [AIs hacking] happening depends a lot on the domain [in question]: How can we codify the rules of the system?"

The key is harnessing AIs for defense, like finding and fixing all vulnerabilities in a software program before it gets released. 

"We'd then live in a world where software vulnerabilities were a thing of the past," he says. 

The downside is the transition period would be vulnerable: Legacy or already-released code could be at risk of attack by AI tools abused by adversaries, he says.

The risk is AI systems hacking other AI systems in the future, and humans experiencing the fallout, he says.

Schneier's latest AI research evolved out of his study of how the hacker mindset and skills could be applied to securing societal systems, which he first presented at the 2020 RSA Conference in San Francisco. This concept, which he coined "hacking society," would mean ethical hackers helping fix the US tax code and legislation to avoid inadvertent or deliberate loopholes, for example.

His big idea boils down to this: "Can we hack society and help secure the systems that make up society?"

Meanwhile, keep an eye on AIs hacking society. 

"Computers are much faster than people. A human process that might take months or years could get compressed to days, hours, or even seconds. What might happen when you feed an AI the entire US tax code and command it to figure out all of the ways one can minimize the amount of tax owed?" he wrote in his essay.

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
I Smell a RAT! New Cybersecurity Threats for the Crypto Industry
David Trepp, Partner, IT Assurance with accounting and advisory firm BPM LLP,  7/9/2021
Attacks on Kaseya Servers Led to Ransomware in Less Than 2 Hours
Robert Lemos, Contributing Writer,  7/7/2021
It's in the Game (but It Shouldn't Be)
Tal Memran, Cybersecurity Expert, CYE,  7/9/2021
Register for Dark Reading Newsletters
White Papers
Current Issue
Everything You Need to Know About DNS Attacks
It's important to understand DNS, potential attacks against it, and the tools and techniques required to defend DNS infrastructure. This report answers all the questions you were afraid to ask. Domain Name Service (DNS) is a critical part of any organization's digital infrastructure, but it's also one of the least understood. DNS is designed to be invisible to business professionals, IT stakeholders, and many security professionals, but DNS's threat surface is large and widely targeted. Attackers are causing a great deal of damage with an array of attacks such as denial of service, DNS cache poisoning, DNS hijackin, DNS tunneling, and DNS dangling. They are using DNS infrastructure to take control of inbound and outbound communications and preventing users from accessing the applications they are looking for. To stop attacks on DNS, security teams need to shore up the organization's security hygiene around DNS infrastructure, implement controls such as DNSSEC, and monitor DNS traffic
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2023-05-26
Craft is a CMS for creating custom digital experiences. Cross site scripting (XSS) can be triggered by review volumes. This issue has been fixed in version 4.4.7.
PUBLISHED: 2023-05-26
Django-SES is a drop-in mail backend for Django. The django_ses library implements a mail backend for Django using AWS Simple Email Service. The library exports the `SESEventWebhookView class` intended to receive signed requests from AWS to handle email bounces, subscriptions, etc. These requests ar...
PUBLISHED: 2023-05-26
Highlight is an open source, full-stack monitoring platform. Highlight may record passwords on customer deployments when a password html input is switched to `type="text"` via a javascript "Show Password" button. This differs from the expected behavior which always obfuscates `ty...
PUBLISHED: 2023-05-26
Craft is a CMS for creating custom digital experiences on the web.The platform does not filter input and encode output in Quick Post validation error message, which can deliver an XSS payload. Old CVE fixed the XSS in label HTML but didn’t fix it when clicking save. This issue was...
PUBLISHED: 2023-05-26
GDSDB infinite loop in Wireshark 4.0.0 to 4.0.5 and 3.6.0 to 3.6.13 allows denial of service via packet injection or crafted capture file