Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


End of Bibblio RCM includes -->
02:45 PM
Connect Directly

When AI Becomes the Hacker

Bruce Schneier explores the potential dangers of artificial intelligence (AI) systems gone rogue in society.

Source: Bruce Schneier
Source: Bruce Schneier

For the past couple of years, renowned technologist and researcher Bruce Schneier has been researching how societal systems can be hacked, specifically the rules of financial markets, laws, and the tax code. That led him to his latest examination of the potential unintended consequences of artificial intelligence on society: how AI systems themselves, which he refers to as "AIs," could evolve such that they automatically - and inadvertently - actually abuse societal systems.

"It's AIs as the hacker," he says, rather than hackers hacking AI systems.

Schneier will discuss his AI hacker research in a keynote address on Monday at the 2021 RSA Conference, which, due to the pandemic, is being held online rather than in person in San Francisco. The AI topic is based on a recent essay he wrote for the Cyber Project and Council for the Responsible Use of AI at the Belfer Center for Science and International Affairs at Harvard Kennedy School.

The core question Schneier asks is this: What if artificial intelligence systems could hack social, economic, and political systems at the computer scale, speed, and range such that humans couldn't detect it in time and suffered the consequences?

It's where AIs evolve into "the creative process of finding hacks."

"They're already doing that in software, finding vulnerabilities in computer code. They're not that good at it, but eventually they will get better [while] humans stay the same" in their vulnerability discovery capabilities, he says. 

In less than a decade from now, Schneier predicts, AIs will be able to "beat" humans in capture-the-flag hacking contests, pointing to the DEFCON contest in 2016 when an AI-only team called Mayhem came in dead last against all-human teams. That's because AI technology will evolve and surpass human capability.

Schneier says it's not so much AIs "breaking into" systems, but AIs creating their own solutions. "AI comes up with a hack and a vulnerability, and then humans look at it and say, 'That's good,'" and use it as a way to make money, like with hedge funds in the financial sector, he says.

The irony here, of course, is that AI starts with human input and programming. Frankenstein analogies aside, the core problem is that AI doesn't have the same human cognitive functions like empathy or a gut check to know where to draw the line. Schneier notes that while there's plenty of research on incorporating context, ethics, and values into AI programs, it's not a built-in function of today's AI systems.

Even so, he says, humans will employ AI to find loopholes in tax codes, such as a major accounting firm doing so to find a new "tax dodge to sell to their customers." So financial firms aren't likely to "program in" rules that thwart their ability to monetize AI knowledge.

The biggest risk is that AIs will find a way around a rule without humans knowing - "that AIs will figure out something that will hack the rules and we won't realize that," Schneier says.

Schneier points to the Volkswagen scandal in 2015, when the carmaker was caught cheating on emissions control-level tests of its vehicle models after engineers programmed the cars' computer systems to activate emissions-curbing only during tests, not in its normal operations. 

"There it was humans attacking the rules" and not AI itself, he says, but it's a good example of what AI could ultimately do to cheat a system if left unchecked to learn ways around it.

In his essay, "The Coming AI Hackers," Schneier describes it this way: "If I asked you to design a car's engine control software to maximize performance while still passing emissions control tests, you wouldn't design the software to cheat without understanding that you were cheating. This simply isn't true for an AI; it doesn't understand the abstract concept of cheating. It will think 'out of the box' simply because it won't have a conception of the box, or of the limitations of existing human solutions. Or of ethics. It won't understand that the Volkswagen solution harms others, that it undermines the intent of the emissions control tests, or that it is breaking the law."

A Wake-up Call to Action
Schneier admits the concept of AIs as hackers is "super speculative" for now, but it's an issue that needs to be addressed. 

"We need to think about this," he says. "And I'm not sure you can stop this. The ease of this [AIs hacking] happening depends a lot on the domain [in question]: How can we codify the rules of the system?"

The key is harnessing AIs for defense, like finding and fixing all vulnerabilities in a software program before it gets released. 

"We'd then live in a world where software vulnerabilities were a thing of the past," he says. 

The downside is the transition period would be vulnerable: Legacy or already-released code could be at risk of attack by AI tools abused by adversaries, he says.

The risk is AI systems hacking other AI systems in the future, and humans experiencing the fallout, he says.

Schneier's latest AI research evolved out of his study of how the hacker mindset and skills could be applied to securing societal systems, which he first presented at the 2020 RSA Conference in San Francisco. This concept, which he coined "hacking society," would mean ethical hackers helping fix the US tax code and legislation to avoid inadvertent or deliberate loopholes, for example.

His big idea boils down to this: "Can we hack society and help secure the systems that make up society?"

Meanwhile, keep an eye on AIs hacking society. 

"Computers are much faster than people. A human process that might take months or years could get compressed to days, hours, or even seconds. What might happen when you feed an AI the entire US tax code and command it to figure out all of the ways one can minimize the amount of tax owed?" he wrote in his essay.

Kelly Jackson Higgins is the Executive Editor of Dark Reading. She is an award-winning veteran technology and business journalist with more than two decades of experience in reporting and editing for various publications, including Network Computing, Secure Enterprise ... View Full Bio

Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
I Smell a RAT! New Cybersecurity Threats for the Crypto Industry
David Trepp, Partner, IT Assurance with accounting and advisory firm BPM LLP,  7/9/2021
Attacks on Kaseya Servers Led to Ransomware in Less Than 2 Hours
Robert Lemos, Contributing Writer,  7/7/2021
It's in the Game (but It Shouldn't Be)
Tal Memran, Cybersecurity Expert, CYE,  7/9/2021
Register for Dark Reading Newsletters
White Papers
Current Issue
The 10 Most Impactful Types of Vulnerabilities for Enterprises Today
Managing system vulnerabilities is one of the old est - and most frustrating - security challenges that enterprise defenders face. Every software application and hardware device ships with intrinsic flaws - flaws that, if critical enough, attackers can exploit from anywhere in the world. It's crucial that defenders take stock of what areas of the tech stack have the most emerging, and critical, vulnerabilities they must manage. It's not just zero day vulnerabilities. Consider that CISA's Known Exploited Vulnerabilities (KEV) catalog lists vulnerabilitlies in widely used applications that are "actively exploited," and most of them are flaws that were discovered several years ago and have been fixed. There are also emerging vulnerabilities in 5G networks, cloud infrastructure, Edge applications, and firmwares to consider.
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
PUBLISHED: 2023-03-27
In Delta Electronics InfraSuite Device Master versions prior to 1.0.5, an attacker could use URL decoding to retrieve system files, credentials, and bypass authentication resulting in privilege escalation.
PUBLISHED: 2023-03-27
In Delta Electronics InfraSuite Device Master versions prior to 1.0.5, an attacker could use Lua scripts, which could allow an attacker to remotely execute arbitrary code.
PUBLISHED: 2023-03-27
Delta Electronics InfraSuite Device Master versions prior to 1.0.5 contains an improper access control vulnerability in which an attacker can use the Device-Gateway service and bypass authorization, which could result in privilege escalation.
PUBLISHED: 2023-03-27
Delta Electronics InfraSuite Device Master versions prior to 1.0.5 are affected by a deserialization vulnerability targeting the Device-DataCollect service, which could allow deserialization of requests prior to authentication, resulting in remote code execution.
PUBLISHED: 2023-03-27
Heap-based Buffer Overflow in GitHub repository gpac/gpac prior to 2.4.0.