Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Security Management

11/27/2019
09:30 AM
Larry Loeb
Larry Loeb
Larry Loeb
50%
50%

False Training Information Can Dupe Machine Learning Models

Researchers from Boston University have shown how really small amounts of disinformation can taint the learning process used by many AI programs.

Researchers from Boston University have recently shown how really small amounts of disinformation can taint the learning process that is used by many "AI" programs.

Panagiota Kiourti, Kacper Wardega, Susmit Jha and Wenchao Li authored the paper that has come out of this effort, "TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents." The paper examines machine learning (ML) systems that are being trained with "reinforcement learning" and came up with a way to fool them so that a Trojan could be slipped into the result of the training.

Neural nets used in ML have long been known to be sensitive to the effects of any low-quality data used in training them. These so-called "adversarial examples" are slightly perturbed inputs that can cause a neural network for a classification task to classify them as a completely different category compared to the original input.

Disturbingly, these perturbed inputs can appear identical to the original from a human perspective.

Sometimes, ML machines will be trained on third-party data sets. Should an attacker gain access to such a model data set and weaponize it with a backdoor to Trojan, the effects could be immense.

The researchers set out to deliberately introduce malicious adversarial examples that would affect the ML's performance in making classifications. For their research, they used a popular and publicly available reinforcement-learning algorithm from DeepMind, called Asynchronous Advantage Actor-Critic, or A3C.

The attack methods were tested on several Atari games that were set up to function in an environment created for reinforcement-learning research. They were Breakout, Pong, Qbert, Space Invaders, Seaquest and Crazy Climber. The games were used since the researchers could measure the effects of the decision/classification performed by the ML used by them.

The attacks are performed on a machine with an Intel i7-6850K CPU and 4×NvidiaGeForce GTX 1080 Ti GPUs that typically completes one training process every 2.4 hours.

Once they tried to defend against attacks they had recognized, things got head-scratching for them. They found that, "Untargeted attacks are difficult to defend against because untargeted attack triggers induce a distribution over outputs […] an effect that breaks the assumptions of Neural Cleanse. There is no demonstrated defense for partial Trojans, where the trigger only corrupts a subset of the output labels."

If an attack is involved with a system having wide dynamic range in its training, they say a defense "will require entirely new defense techniques as all known defenses rest on the basis of discrete outputs. Furthermore, we claim that previous works promising defenses under Threat Model 2 are not effective on Trojaned DRL agents as large training sets and small amount of poisoned inputs inhibit the proper function of such techniques."

So, they can get ML systems to make major classification errors with these adversarial examples, but they are not sure how to defend against them. It makes sense for them to conclude that, "Our work suggests caution in deploying reinforcement learning in high-security safety-critical applications where the training process is not restricted to a controlled and secure environment."

— Larry Loeb has written for many of the last century's major "dead tree" computer magazines, having been, among other things, a consulting editor for BYTE magazine and senior editor for the launch of WebWeek.

Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
Edge-DRsplash-10-edge-articles
I Smell a RAT! New Cybersecurity Threats for the Crypto Industry
David Trepp, Partner, IT Assurance with accounting and advisory firm BPM LLP,  7/9/2021
News
Attacks on Kaseya Servers Led to Ransomware in Less Than 2 Hours
Robert Lemos, Contributing Writer,  7/7/2021
Commentary
It's in the Game (but It Shouldn't Be)
Tal Memran, Cybersecurity Expert, CYE,  7/9/2021
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
How Enterprises are Attacking the Cybersecurity Problem
Concerns over supply chain vulnerabilities and attack visibility drove some significant changes in enterprise cybersecurity strategies over the past year. Dark Reading's 2021 Strategic Security Survey showed that many organizations are staying the course regarding the use of a mix of attack prevention and threat detection technologies and practices for dealing with cyber threats.
Flash Poll
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2021-3454
PUBLISHED: 2021-10-19
Truncated L2CAP K-frame causes assertion failure. Zephyr versions >= 2.4.0, >= v.2.50 contain Improper Handling of Length Parameter Inconsistency (CWE-130), Reachable Assertion (CWE-617). For more information, see https://github.com/zephyrproject-rtos/zephyr/security/advisories/GHSA-fx88-6c29-...
CVE-2021-3455
PUBLISHED: 2021-10-19
Disconnecting L2CAP channel right after invalid ATT request leads freeze. Zephyr versions >= 2.4.0, >= 2.5.0 contain Use After Free (CWE-416). For more information, see https://github.com/zephyrproject-rtos/zephyr/security/advisories/GHSA-7g38-3x9v-v7vp
CVE-2021-41150
PUBLISHED: 2021-10-19
Tough provides a set of Rust libraries and tools for using and generating the update framework (TUF) repositories. The tough library, prior to 0.12.0, does not properly sanitize delegated role names when caching a repository, or when loading a repository from the filesystem. When the repository is c...
CVE-2021-31378
PUBLISHED: 2021-10-19
In broadband environments, including but not limited to Enhanced Subscriber Management, (CHAP, PPP, DHCP, etc.), on Juniper Networks Junos OS devices where RADIUS servers are configured for managing subscriber access and a subscriber is logged in and then requests to logout, the subscriber may be fo...
CVE-2021-31379
PUBLISHED: 2021-10-19
An Incorrect Behavior Order vulnerability in the MAP-E automatic tunneling mechanism of Juniper Networks Junos OS allows an attacker to send certain malformed IPv4 or IPv6 packets to cause a Denial of Service (DoS) to the PFE on the device which is disabled as a result of the processing of these pac...