Manipulated Machine Learning Sows Confusion

Machine learning, and neural networks in particular, will become a prime target for those aiming to manipulate or disrupt dependent products and services. Attackers will exploit vulnerabilities and flaws in machine learning systems by confusing and deceiving algorithms in order to manipulate outcomes for nefarious purposes.

Impacts will be felt across a range of industries. Malicious attacks may result in automated vehicles changing direction unexpectedly, high-frequency trading applications making poor financial decisions and airport facial recognition software failing to recognize terrorists. Organizations will face significant financial, regulatory and reputational damage and lives will be put at risk if machine learning systems are compromised.

Nation states, terrorists, hacking groups, hacktivists and even rogue competitors will turn their attention to manipulating machine learning systems that underpin products and services. Attacks that are undetectable by humans will target the integrity of information -- widespread chaos will ensue for those dependent on services powered primarily by machine learning.

What is the justification for this threat?
A range of industries will increasingly adopt machine learning systems and neural networks over the coming years in order to help make faster, smarter decisions. They will be embedded into a series of business operations such as marketing, medicine, retail, automated vehicles and military applications. The explosion of data from connected sensors, IoT devices and social media outputs will drive companies to use machine learning to automate processes, with minimal human oversight. As these technologies begin to underpin business models, they will become a prime target.

Academics have already provided several proof of concept studies highlighting how machine learning can be confused. Students at MIT developed a means of tricking neural network-based image recognition software built on Google’s open source software library TensorFlow, forcing the neural network to miscategorize an image of a turtle as a gun. The real-world implications of deliberately miscategorizing images used as inputs to machine learning systems would transcend all industries that adopt machine learning for processes and services, significantly threatening human life and damaging corporate reputations.

Attackers can place ‘noise’ over an image or sound, which is undetectable by humans but is picked up by the neural network. This noise fools the neural network into miscategorizing the image it sees. Academics could, for example, fool a neural network in a connected vehicle causing it to miscategorize a stop sign as a different road sign. By spraying graffiti over the stop sign, the neural network is unable to categorize it correctly; whereas the human eye could make assumptions, even though the sign had been vandalized. If connected vehicles were fooled in this way, there would be significant risk to human life.

Neural networks used in connected vehicles typically gain significant media scrutiny. In March of 2018, a self-driving Uber vehicle travelling at 40mph in Tempe, Arizona, fatally struck a pedestrian crossing the street in the dark as a result of the vehicle’s perception system miscategorizing a bicycle she was wheeling as something else. There have also been two cases where image recognition software in Tesla vehicles failed to categorize fire trucks and other vehicles, causing a fatality in one case. While these examples were accidental, they highlight the potential for malicious attacks.

How should your organization prepare?
The damage a compromised machine learning system may bring could be life threatening. Organizations should assess their offerings and dependency on machine learning systems before attackers exploit related vulnerabilities.

Short-term actions include identifying systems which use machine learning, especially recognition-based neural networks, and determine their criticality to the business, employing technical experts in machine learning and recognition-based neural networks and gaining assurance that machine learning and recognition-based neural networks provided by external parties are secure by design. In the longer term, continuously monitor the dark web for vulnerabilities and exploit code, and lobby governments and regulators to introduce regulation around reporting requirements, demanding clear outlines of usage and protection of data used by algorithms in machine learning systems.

— Steve Durbin is managing director of the Information Security Forum (ISF). His main areas of focus include the emerging security threat landscape, cybersecurity, BYOD, the cloud and social media across both the corporate and personal environments. Previously, he was senior vice president at Gartner.