Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Operational Security //

AI

3/15/2018
09:35 AM
Dawn Kawamoto
Dawn Kawamoto
Dawn Kawamoto
50%
50%

AI: An Emerging Insider Threat?

As artificial intelligence increasingly gains a presence in the enterprise, concerns are already being raised of a new insider threat where AI will turn against its operators. How can security experts address this "frenemy"?

Machines being smarter than humans has already played out across national TV and the Internet, from IBM's supercomputer Watson defeating renowned Jeopardy! game contestants to Google's AlphaGo beating the world's best Go game player.

And while the closely watched Jeopardy and Go competitions demonstrated how computers raised on machine learning and packed with artificial intelligence can surpass human intelligence and hold vast potential for delivering good to society, concerns have also emerged about the darker side of the technology.

Visionary Elon Musk, for one, has described AI as "summoning the demon" and it could be humanity's "biggest existential threat," according to a video posting in the Washington Post of his presentation at MIT.

And Google DeepMind, creator of AlphaGo, and Stuart Armstrong, an Alexander Tamas Programme on AI Safety research fellow at Oxford University, warned of the need for a "red button" to disable an AI agent if it learned to avoid being interrupted or stopped by its operator in their report, "Safely Interruptible Agents."

Should IT security professionals be worried about machine learning and AI potentially becoming the new insider threat?

AI researchers, including Armstrong and Shahar Avin, a research associate with Cambridge University's Centre for the Study of Existential Risk and co-author of "The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation," and Sam Bouso, CEO of Precognitive, an IT security company that uses machine learning and AI, weigh in on this issue.

Near-term outlook
AI is not likely to emerge as a new intentional insider threat in the near term, experts say, pointing to several factors at play.

"I don't see why AI is specifically an insider threat. In the short term, AI could amplify the capabilities of bad actors and possibly of good actors, whether or not they are insiders or not. In the medium term, AI could automate social engineering approaches such as spear phishing, but that's not specifically an insider threat either," Armstrong told Security Now.

Bouso told Security Now that AI is decades away from becoming a credible insider threat, noting that currently the technology is not at a point where it can think on its own and decide between good and bad.

"AI and ML are certainly being used for cybercrime but there is someone behind it when it happens, and they have a specific intent to do harm," Bouso said. "Any sort of rogue AI is nothing more than bad programming and human error at this point."

Indeed. Avin, in an interview with Security Now, noted that while machine learning and AI systems currently do not have a "will" of their own, it is up to companies to set the right policies and restrictions in place to guide the machine's learning. Unfortunately, that may prove challenging.

"We don't yet have robust tools to fully reason through all possible policies the system might take, and so [IT security professionals] should be extra careful with the inputs and outputs connected to such systems, and adversarially test them through red teaming efforts," Avin advised.

The "black-box" nature of many contemporary AI and machine learning algorithms, coupled with their ability to discover novel behaviors or policies within an environment or goal, should prompt companies to give additional care when introducing them to corporate networks and systems -- especially those that contain sensitive or confidential information, Avin noted.

And although IT security folks are concerned and discuss AI and its potential for self-awareness and malicious intent, Bouso said it's viewed along the lines of "one day this could happen."

Long-term outlook
Although cyber attackers are currently using machine learning and AI to wage war on companies and consumers, future development of the technology may one day lead to these technologies becoming IT security professionals' "frenemies."

"Deploying malicious code seems to be an extreme end, and unlikely until we have much more capable systems which are much more trusted," Avin said. "However, it is not unreasonable that a Q&A system will disclose information it was not intended to disclose, or, perhaps more worryingly, an automated configuration generator will create configs that exploit edge cases in the system being configured, such that a metric is maximized, but the system itself fails."

Researchers are focused on further machine learning and AI enhancements to make capable systems that are more trusted, a move the security industry is clamoring for as it faces a steep shortage of workers and fatigue from a constant barrage of alerts from their security operations center (SOC).

Development is underway for artificial general intelligences (AGIs), as an example.

"Before we get true AGIs, I expect any bugs created by AIs to be minor and non-deliberate. If and when AIs become AGIs, then I expect them to have enough knowledge and learning abilities to deploy malicious code if motivated to do that -- it would have most of the capabilities of humans, and be better in many areas," Armstrong said. "But as I said before, if we assume true AGIs, then civilization will be radically changed anyway."

Related posts:

— Dawn Kawamoto is an award-winning technology and business journalist, whose work has appeared in CNET's News.com, Dark Reading, TheStreet.com, AOL's DailyFinance, and The Motley Fool.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
COVID-19: Latest Security News & Commentary
Dark Reading Staff 9/25/2020
Hacking Yourself: Marie Moe and Pacemaker Security
Gary McGraw Ph.D., Co-founder Berryville Institute of Machine Learning,  9/21/2020
Startup Aims to Map and Track All the IT and Security Things
Kelly Jackson Higgins, Executive Editor at Dark Reading,  9/22/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Special Report: Computing's New Normal
This special report examines how IT security organizations have adapted to the "new normal" of computing and what the long-term effects will be. Read it and get a unique set of perspectives on issues ranging from new threats & vulnerabilities as a result of remote working to how enterprise security strategy will be affected long term.
Flash Poll
How IT Security Organizations are Attacking the Cybersecurity Problem
How IT Security Organizations are Attacking the Cybersecurity Problem
The COVID-19 pandemic turned the world -- and enterprise computing -- on end. Here's a look at how cybersecurity teams are retrenching their defense strategies, rebuilding their teams, and selecting new technologies to stop the oncoming rise of online attacks.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-15208
PUBLISHED: 2020-09-25
In tensorflow-lite before versions 1.15.4, 2.0.3, 2.1.2, 2.2.1 and 2.3.1, when determining the common dimension size of two tensors, TFLite uses a `DCHECK` which is no-op outside of debug compilation modes. Since the function always returns the dimension of the first tensor, malicious attackers can ...
CVE-2020-15209
PUBLISHED: 2020-09-25
In tensorflow-lite before versions 1.15.4, 2.0.3, 2.1.2, 2.2.1 and 2.3.1, a crafted TFLite model can force a node to have as input a tensor backed by a `nullptr` buffer. This can be achieved by changing a buffer index in the flatbuffer serialization to convert a read-only tensor to a read-write one....
CVE-2020-15210
PUBLISHED: 2020-09-25
In tensorflow-lite before versions 1.15.4, 2.0.3, 2.1.2, 2.2.1 and 2.3.1, if a TFLite saved model uses the same tensor as both input and output of an operator, then, depending on the operator, we can observe a segmentation fault or just memory corruption. We have patched the issue in d58c96946b and ...
CVE-2020-15211
PUBLISHED: 2020-09-25
In TensorFlow Lite before versions 1.15.4, 2.0.3, 2.1.2, 2.2.1 and 2.3.1, saved models in the flatbuffer format use a double indexing scheme: a model has a set of subgraphs, each subgraph has a set of operators and each operator has a set of input/output tensors. The flatbuffer format uses indices f...
CVE-2020-15212
PUBLISHED: 2020-09-25
In TensorFlow Lite before versions 2.2.1 and 2.3.1, models using segment sum can trigger writes outside of bounds of heap allocated buffers by inserting negative elements in the segment ids tensor. Users having access to `segment_ids_data` can alter `output_index` and then write to outside of `outpu...