Architectural Analysis IDs 78 Specific Risks in Machine-Learning Systems

Researchers at the Berryville Institute of Machine Learning (BIML) have developed a formal risk framework to guide development of secure machine-language (ML) systems.

BIML's architectural risk analysis of ML systems is different from previous work in this area in that it focuses on issues that engineers and developers need to be paying attention to at the outset when designing and building ML systems. Most of the previous work on securing ML systems has focused on how to best protect operational systems and data against particular attacks and not on how to design them securely in the first place.

"This work provides a very solid technical foundation for taking a look at the risks associated with adopting and using ML," says Gary McGraw, noted security researcher, author, and co-founder of BMIL. The need for this kind of a risk analysis is critical because very few are really paying any attention to ML security at the design state, even as ML use is growing rapidly, he says.

For the architectural risk analysis, BIML researchers considered nine separate components that they identified as common to setting up, training, and deploying a typical ML system: raw data; dataset assembly; datasets; learning algorithms; evaluation; inputs; trained model; inference algorithm; and outputs. They then identified and ranked multiple data security risks associated with each of those components so engineers and developers can implement controls for mitigating those risks where possible.

For instance, they identified data confidentiality, the trustworthiness of data sources, and data storage as key security considerations around the raw data used in ML systems, such as training data, test inputs, and operational data. Similarly, for the datasets used in ML systems, the researchers identified data poisoning — where an attacker manipulates data to cause ML systems to go awry — as a major risk. For training algorithms, BIML researchers identified the potential for attackers to subtly nudge an online learning system in a direction not intended by its developers as a major concern.

In total, BIML's architectural analysis showed that typical ML systems are exposed to as many as 78 specific security risks across all individual components. They categorized the risks under multiple categories including input manipulation, data manipulation, model manipulation, and extraction attacks where threat actors try and extract sensitive data from an ML system dataset.

McGraw says the BIML analysis is about identifying and discussing ML risks and discussing them, and not so much about what to do about them. "Identifying the risks is more than half the battle," he says. "Once you know what the risks are, it's a lot easier to design around them."

The BMIL report listed the top 10 risks impacting ML systems. According to the think tank, the biggest — and most commonly discussed risks — to ML systems are so-called "adversarial examples" involving the use of malicious inputs to cause the system to make false predictions or categorizations. Data poisoning, online system manipulation, and attacks impacting data confidentiality, data integrity, and data output were all identified as other top ML security risks.

The Importance of Data Security
"One of the remarkable differences in ML security and, say, normal operational security is that data and data security play a huge role," says McGraw. "When you are training up a system, you can train it up to be racist and xenophobic and horrible if your data are set up that way," he says.

As one example, he points to Microsoft's very short-lived experiment with Tay, an AI-enabled chatbot that learned from interactions on Twitter and quickly began spewing out venomous tweets of its own. "Tay was learning about Twitter by being on it, and what happened was it became a racist, bigoted troll," he says. "Tay learned what it was like to be on Twitter, and it wasn't pretty."  

Such incidents highlight why organizations need to think carefully about the data they are using for machine training, how the data gets sourced, and whether the sources are reliable, he says.

Contrary to what some might assume, attacking a machine-learning system is not all that complicated, McGraw notes. "Imagine the input data for Google Translate is anything that you type in," he says. "If you are using public data sources to train your machine learning model, you have to think about what happens when an attacker starts screwing around with.

"The good news is if you are an engineer or a designer, you can make it harder for someone to attack your system. That's the purpose of this work."

Related Content:

Check out The Edge, Dark Reading's new section for features, threat data, and in-depth perspectives. Today's featured story: "Chaos & Order: The Keys to Quantum-Proof Encryption"