Better Cybersecurity Research Requires More Data Sharing

Good data on attacker tactics, security incidents, and breaches is key to identifying trends in cybersecurity, but datasets — even among academic researchers — are often not made public and just as often are of poor quality, according to security researchers who presented their conclusions at the annual Workshop on the Economics of Information Security (WEIS) conference.  

In a previous sampling of some 965 papers, a group of researchers from the University of Tulsa found that only 6% created their own datasets and made them public. Yet the value of such data exceeds $663 million just in cost savings to subsequent research efforts, according to a paper presented at the WEIS conference by the same group.

"There is a lot of value there," says Tyler Moore, one of the authors and an associate professor of cybersecurity at the Tandy School of Computer Science at the University of Tulsa. "Cybersecurity research datasets do create a lot of value by making the data available to other researchers."

Data is key to a variety of initiatives in cybersecurity. From training machine-learning systems to detect threats to analyzing whether breach regulations actually results in better defenses, researchers and security professionals need more and better datasets. 

Yet sharing of data — even among researchers — does not happen often enough, says Sam Ransbotham, co-chairman of the WEIS 2019 conference and associate professor of the Carroll School of Management at Boston College.

"Everyone wants to use these datasets, but no one wants to create them and release them," he says. "Companies want to show you all the cool stuff that they have done, but ask for their security data and that is a problem."

Seven years after their original research into the costs of breaches, a group of researchers from the University of Cambridge pulled together the best data on current breach costs. The data is very scattershot and authoritative information is still not available, but the researchers are doing the best with the information they can obtain, Ransbotham says.

"The problem, trying to get more data, goes throughout all the papers," he says. "None of this data is perfect. If you look at almost everything they talked about — and this is not negative — it is the current state of uncertainty."

The value of the data is significant, the University of Tulsa researchers found. In the paper they presented at the WEIS conference, Moore and the other researchers analyzed a collection of cybersecurity datasets funded by the US Department of Homeland Security (DHS) and known as IMPACT; they found that almost 2,300 people have requested data from the system. With the average dataset valued at $291,000 in saved costs, those requests total $663 million. 

"The research community has to decide what our norms will be and what our values are," Moore says. "There should be a default to make data sets available in order to view research as a scientific contribution."

In previous research, Moore and other University of Tulsa researchers found that of a sample of 965 research papers, 55% used data in some form. About 44% of those papers using data did not create their own dataset, while 56% did. Yet only 6% of 965 papers both created their own data and publicly shared that data, according to the University of Tulsa research.

That leaves many researchers struggling to find creative ways to use existing data sets to produce new insights, says Boston College's Ransbotham.

"These are people using trace data in some new and clever ways," he says. "They are trying to figure out stuff from the data that is leaked out there in ways that are new."

For many researchers, the problem is that sharing data gives no significant payoff. However, the University of Tulsa team found that a slight benefit: Research papers that included public datasets were cited an additional four times per year, which is small but significant, Tyler says. 

Still, that may not be enough to convince many researchers to share, he acknowledges.

"There needs to be a recognition in the community to share more widely, but how do we get there?" he says. "It would be great to have benefactors because right now you only have a very small amount of money from DHS."

Related Content: