Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats //

Advanced Threats

11/2/2020
06:50 PM
50%
50%

Microsoft & Others Catalog Threats to Machine Learning Systems

Thirteen organizations worked together to create a dictionary of techniques used to attack ML models and warn that such malicious efforts will become more common.

In May 2016, Microsoft introduce a chatbot on Twitter, dubbed "Tay," that attempted to hold conversations with users and improve its responses through machine learning (ML). A coordinated attack on the chatbot, however, caused the algorithm to start tweeting "wildly inappropriate and reprehensible words and images" in the first 24 hours, Microsoft stated at the time.

For the software giant, the attack demonstrated that the world of ML and artificial intelligence (AI) would come with threats. Last week, the company and an interdisciplinary group of security professionals and ML researchers from a dozen other organizations took a first stab at creating a vocabulary for describing attacks on ML systems with the initial draft of the Adversarial ML Threat Matrix.

The threat matrix is an extension of MITRE's ATT&CK framework for the classification of attack techniques. The information should help secure not just the developers of ML systems but companies that are using those systems as well, says Jonathan Spring, senior member of the technical staff of the CERT Division of Carnegie Mellon University's Software Engineering Institute.

Related Content:

Using Adversarial Machine Learning, Researchers Look to Foil Facial Recognition

The Changing Face of Threat Intelligence

New on The Edge: How Can I Help Remote Workers Secure Their Home Routers?

"If you're using a machine learning system — even if you're not the one developing it — you should make sure that your broader system is fault tolerant," Spring says. "You should be looking for people pressing on [attacking] the broader machine learning part of your system. And you can do those checks on your system without really knowing too much about how the machine learning is working."

Machine learning has become a key factor in companies' plans to transform their businesses over the next decade. Yet, most firms consider adversarial attacks on ML to be a future threat, not a current risk. Only three of 28 companies surveyed by Microsoft, for example, thought they had the tools in place to secure their ML systems. 

Actual attacks on ML systems inhabit a spectrum of generic exploits of vulnerabilities to specific ML-reliant attacks on models or data. In one case, an attacker exploited a misconfiguration in the system of the facial recognition firm ClearviewAI to gain access to some of its infrastructure, which could have resulted in the attacker polluting the dataset.

"[W]e believe the first step in empowering security teams to defend against attacks on ML systems, is to have a framework that systematically organizes the techniques employed by malicious adversaries in subverting ML systems," Microsoft's researchers said in a blog post announcing the Adversarial ML Threat Matrix. "We hope that the security community can use the tabulated tactics and techniques to bolster their monitoring strategies around their organization's mission-critical ML systems."

The Adversarial ML Threat Matrix is based on the MITRE ATT&CK framework, which has grown in popularity since it was originally released in 2015. More than 80% of companies use the framework as part of their security response programs, according to an October survey published by the University of California at Berkeley and McAfee in October.

The threat matrix is the work of a baker's dozen of different organizations. Microsoft, Carnegie Mellon University's Software Engineering Institute, and MITRE are collaborating with Bosch, IBM, NVIDIA, Airbus, Deep Instinct, Two Six Labs, the University of Toronto, Cardiff University, PricewaterhouseCoopers, and Berryville Institute of Machine Learning on the framework. The team used a variety of case studies to identify the common tactics and techniques used by attackers and describe them for security researchers. 

At the DerbyCon conference in 2019, for example, two researchers showed a way to use a data-based attack against Proofpoint's email security system to extract the training data and create a system that could be used by an attacker to as a test platform for creating email attacks that would not be caught by the messaging security product. Microsoft also mined its experience with the Tay chatbot to inform the threat matrix.

While the risks to ML and AI systems are real, they aren't the most common threats, Charles Clancy, chief futurist and general manager of MITRE Labs, said in an interview. "Typically, AI isn’t the first avenue for our adversaries, particularly regarding attacking our critical infrastructure," he said. "There's a truism in the power industry that the most dangerous adversaries to our electric grid are — squirrels. Keep that in mind — there are risks to AI, but it's also extremely valuable."

The Adversarial ML Threat Matrix is only the first attempt to capture all the threats posed to ML systems. The companies and security researchers called for others to contribute their experiences as well. 

"Perhaps this first version of the Adversarial ML Threat Matrix captures the adversary behavior you have observed — [i]f not, please contribute what you can to MITRE and Microsoft so your experience can be captured," CMU's Software Engineering Institute stated in its blog post. "If the matrix does reflect your observations, is it helpful in communicating and understanding this adversary behavior and explaining threats to your constituents? Share those experiences with the authors as well, so the matrix can improve!"

Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio
 

Recommended Reading:

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 11/19/2020
New Proposed DNS Security Features Released
Kelly Jackson Higgins, Executive Editor at Dark Reading,  11/19/2020
The Yellow Brick Road to Risk Management
Andrew Lowe, Senior Information Security Consultant, TalaTek,  11/19/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win an Amazon Gift Card! Click Here
Latest Comment: He hits the gong anytime he sees someone click on an email link.
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you today!
Flash Poll
The State of Ransomware
The State of Ransomware
Ransomware has become one of the most prevalent new cybersecurity threats faced by today's enterprises. This new report from Dark Reading includes feedback from IT and IT security professionals about their organization's ransomware experiences, defense plans, and malware challenges. Find out what they had to say!
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-14190
PUBLISHED: 2020-11-25
Affected versions of Atlassian Fisheye/Crucible allow remote attackers to achieve Regex Denial of Service via user-supplied regex in EyeQL. The affected versions are before version 4.8.4.
CVE-2020-29074
PUBLISHED: 2020-11-25
scan.c in x11vnc 0.9.16 uses IPC_CREAT|0777 in shmget calls, which allows access by actors other than the current user.
CVE-2020-14191
PUBLISHED: 2020-11-25
Affected versions of Atlassian Fisheye/Crucible allow remote attackers to impact the application's availability via a Denial of Service (DoS) vulnerability in the MessageBundleResource within Atlassian Gadgets. The affected versions are before version 4.8.4.
CVE-2020-29070
PUBLISHED: 2020-11-25
osCommerce 2.3.4.1 has XSS vulnerability via the authenticated user entering the XSS payload into the title section of newsletters.
CVE-2020-26212
PUBLISHED: 2020-11-25
GLPI stands for Gestionnaire Libre de Parc Informatique and it is a Free Asset and IT Management Software package, that provides ITIL Service Desk features, licenses tracking and software auditing. In GLPI before version 9.5.3, any authenticated user has read-only permissions to the planning of ever...