Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Threat Intelligence

Adobe Open Sources Tool for Anomaly Research

The One-Stop Anomaly Shop (OSAS) project packages machine-learning algorithms into a Docker container for finding anomalies in security log data.

Adobe has released an open source project to detect and classify anomalies in security log data using a tool the company says is simple to run and easily modified. 

The One-Stop Anomaly Shop (OSAS) is an open source machine-learning (ML) tool that can add structure to log data by generating labels for different types of data and then use that data as the inputs to classification algorithms. The approach solves several ML problems — such as data sparsity and overfitting — while giving security analysts a macro view of log data that allows easier analysis, states Adobe's security intelligence team in a technical paper on their approach.

Related Content:

Microsoft Uses Machine Learning to Predict Attackers' Next Steps

Special Report: Assessing Cybersecurity Risk in Today's Enterprises

New From The Edge: 10K Hackers Defend the Planet Against Extraterrestrials

The software project allows security teams to quickly identify what features — or feature combinations — provide the most benefit in terms of analysis in a particular dataset, the Adobe Security Intelligence Team told Dark Reading in an email interview. 

"One could consider finding anomalies as somewhat trivial from the computational perspective," they said. "However, being able to say why something is an anomaly is an entirely different story. OSAS is useful in identifying why [an event is considered] an anomaly."

The One-Stop Anomaly Shop project provides security analysts and researchers with a way to quickly analyze security logs using labeled data, even when the security log file has a variety of event types, the Adobe group stated in a blog post. The project, available on GitHub, creates a Docker container running the Elasticsearch search engine, Logstash indexer, and Kibana Web front-end — a combination known as the ELK stack — while the ML application is written in Python.

The system labels events with a variety of tags, indicating, for example, whether the anomaly is unique, whether a particular port, process, or path is rare, and whether the event connects to a public IP or the localhost. 

"There is a lot of research and whitepapers on data-science in security, but few tools that implement state-of-the-art ideas that are made available to the community," Adobe's team told Dark Reading. "Primarily, open-sourcing OSAS was an opportunity for us to put our work in an end-to-end framework. Secondly, we want to make OSAS as robust and security oriented as possible and we cannot achieve that without support from the security community."

Adobe is not the only software company to provide security teams with ML tools. Earlier this month, Microsoft published details of a project that uses the company's massive data set of attack traffic, along with the MITRE ATT&CK framework, to build an ML model that not only assigns particular attack tactics with certain groups, but predicts the attacker's potential next steps.

Adobe's tool is best-suited for working with security log data, but it can work on any source of flat log data that follow the same patterns, such as authentication logs, Web server logs, and access logs, Adobe stated.

After tagging the various elements of the log file, a second pipeline also assigns risk-based scores to collections of tags.

"The primary goal is to assign high scores to suspicious activity and low scores to normal operations," the security intelligence team states in its technical whitepaper.

The ML tool has multiple strategies for detecting anomalies in log files and assigning them risk-based scores. An unsupervised-learning approach may find malicious activities that would go undiscovered with supervised-learning models but will also likely generate more false alerts. 

"In theory, potentially malicious events are a subset of the anomalies set," the security intelligence team said via email. "Targeting the detection of these potentially malicious events by OSAS can be achieved by creating a tailored data-grooming pipeline via the configuration file."

In tests, the security intelligence team used benign data of normal operations to train the ML algorithm and then input an artificially constructed dataset to benchmark the ability to detect anomalies. The supervised approach had a nearly 95% detection rate, while two unsupervised models performed less well, with a 63% and 50% score.

The Adobe Security Intelligence Team aims to garner feedback with the project and perhaps build a collection of pretrained analysis pipelines — a "model-zoo" — that can be distributed with future versions. The team comprises security engineers Vivek Malik and Kumar Vikramjeet, data scientist Tiberiu Boros, and technical lead Andrei Cotaie.

Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
Edge-DRsplash-10-edge-articles
I Smell a RAT! New Cybersecurity Threats for the Crypto Industry
David Trepp, Partner, IT Assurance with accounting and advisory firm BPM LLP,  7/9/2021
News
Attacks on Kaseya Servers Led to Ransomware in Less Than 2 Hours
Robert Lemos, Contributing Writer,  7/7/2021
Commentary
It's in the Game (but It Shouldn't Be)
Tal Memran, Cybersecurity Expert, CYE,  7/9/2021
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
Incorporating a Prevention Mindset into Threat Detection and Response
Threat detection and response systems, by definition, are reactive because they have to wait for damage to be done before finding the attack. With a prevention-mindset, security teams can proactively anticipate the attacker's next move, rather than reacting to specific threats or trying to detect the latest techniques in real-time. The report covers areas enterprises should focus on: What positive response looks like. Improving security hygiene. Combining preventive actions with red team efforts.
Flash Poll
How Enterprises are Developing Secure Applications
How Enterprises are Developing Secure Applications
Recent breaches of third-party apps are driving many organizations to think harder about the security of their off-the-shelf software as they continue to move left in secure software development practices.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2022-1678
PUBLISHED: 2022-05-25
An issue was discovered in the Linux Kernel from 4.18 to 4.19, an improper update of sock reference in TCP pacing can lead to memory/netns leak, which can be used by remote clients.
CVE-2021-32966
PUBLISHED: 2022-05-25
Philips Interoperability Solution XDS versions 2.5 through 3.11 and 2018-1 through 2021-1 are vulnerable to clear text transmission of sensitive information when configured to use LDAP via TLS and where the domain controller returns LDAP referrals, which may allow an attacker to remotely read LDAP s...
CVE-2021-32989
PUBLISHED: 2022-05-25
When a non-existent resource is requested, the LCDS LAquis SCADA application (version 4.3.1.1011 and prior) returns error messages which may allow reflected cross-site scripting.
CVE-2021-32997
PUBLISHED: 2022-05-25
The affected Baker Hughes Bentley Nevada products (3500 System 1 6.x, Part No. 3060/00 versions 6.98 and prior, 3500 System 1, Part No. 3071/xx & 3072/xx versions 21.1 HF1 and prior, 3500 Rack Configuration, Part No. 129133-01 versions 6.4 and prior, and 3500/22M Firmware, Part No. 288055-01 ver...
CVE-2021-35487
PUBLISHED: 2022-05-25
Nokia Broadcast Message Center through 11.1.0 allows an authenticated user to perform a Boolean Blind SQL Injection attack on the endpoint /owui/block/send-receive-updates (for the Manage Alerts page) via the extIdentifier HTTP POST parameter. This allows an attacker to obtain the database user, dat...