Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Attacks/Breaches

3/30/2016
10:30 AM
Commentary
Commentary
Commentary
50%
50%

Machine Learning In Security: Good & Bad News About Signatures

Why security teams that rely solely on signature-based detection are overwhelmed by a high number of alerts.

First in a series of two articles about the history of signature-based detections, and how the methodology has evolved to identify different types of cybersecurity threats.

Used in the context of an outdated and manually intensive technology focused on older classes of threats, there’s little wonder why vendors would seek to distance the legacy term “signature” from their advanced detection technology. Vendors haven’t necessarily been deceptive in the labeling of their latest generation of techniques; it’s often just easier to create a new label for something than to fully explain the context and evolution of what preceded it.

Over the years, signature-based systems have changed and advanced, but the core concepts still lie at the heart of all modern detection systems – and will continue to be integral for the foreseeable future. To understand what a “signature system” is in reality, we need to understand the evolution of the detection path as directed and discovered by human intervention.

One-dimensional signatures: Blacklists and whitelists are examples of one-dimensional signature systems. They are found throughout security and exist in practically all detection and protection technologies. They are by far the fastest and most efficient way of categorizing a data artifact (e.g. a domain name, IP address, user-agent, MD5 hashes, etc.). As a Boolean operation, what you’re looking for is either on the list or it is not.

Two-dimensional signatures: Classic regular-expression functions and string matching are examples of two-dimensional signature systems. They are the fundamental building blocks of anti-malware, intrusion detection, and data leakage detection systems. In malware, they are often used to search a binary file for known strings which help to label the type of threat it represents. Two-dimensional signatures came to the fore as a means of detecting network-based threats within the content-level of traffic – easily capable of identifying previously known exploits and host enumeration techniques.

Data leakage prevention (DLP) is a more recent security technology that relies heavily upon two-dimensional signatures. Messages and file attachments are often scanned for specific strings (e.g. serial numbers, passwords, etc.) or construction formats (e.g. social security numbers of the format nnn-nn-nnnn with a regular expression of ^\d{3}-\d{2}-\d{4}$ ). 

Multidimensional signatures: Security vendors developed a hybrid system as the threat spectrum grew and attackers found new ways to obfuscate the elements of their attacks that were most exposed to one-dimensional and two-dimensional signatures. Instead of triggering on a single signature, a multi-dimensional signature was created. In both sandboxing and network behavioral monitoring, certain actions and activities are labeled as either suspicious or bad. 

When a threshold of good or bad activities is reached, the threat is classified and labeled. For example, a suspicious file is executed within a virtual environment. The file attempts to write to the Windows registry (neither good nor bad), add a file to the Windows startup path (suspicious), disable Windows updates (bad), read from the user’s contacts list (neither good nor bad), and then send email to every address listed in the contacts list (bad).

Together, all of these individual actions (i.e. signatures) are combined and tallied and a decision is made that the suspicious file is in fact malicious and most likely a spambot.

Signature systems all share the same characteristic of being able to promptly identify and label a threat. As signature systems have evolved, they have become capable of detecting and classifying a broader range of threats. In modern detection and prevention systems, a combination of different signature systems are used together so they can most accurately label a known threat, but this also has the problem of generating a high number of alerts that can overwhelm a team that solely relies on signature-based detection for security purposes. 

Historically, the linear progression and sophistication of signature-based detection systems have been dependent upon human signature writers. For each new threat, a unique signature or signature artifact is created by a skilled engineer or security researcher. This pairing between signature and its human creator means that as the number of threats have increased, so too have the number of skilled personnel needed to develop and support the signatures that detect them. For obvious reasons, this is not a scalable business proposition – for neither the vendor or customer.

New developments in machine learning – in particular supervised and unsupervised learning algorithms – are now being applied to information security and are paving the way to a new class of signature systems capable of economically scaling to the threat. 

Next in the series: Machine Learning In Security: Seeing the Nth Dimension in Signatures 

Related Content: 

 

Interop 2016 Las VegasFind out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Click here for pricing information and to register.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Stop Defending Everything
Kevin Kurzawa, Senior Information Security Auditor,  2/12/2020
Small Business Security: 5 Tips on How and Where to Start
Mike Puglia, Chief Strategy Officer at Kaseya,  2/13/2020
5 Common Errors That Allow Attackers to Go Undetected
Matt Middleton-Leal, General Manager and Chief Security Strategist, Netwrix,  2/12/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
6 Emerging Cyber Threats That Enterprises Face in 2020
This Tech Digest gives an in-depth look at six emerging cyber threats that enterprises could face in 2020. Download your copy today!
Flash Poll
How Enterprises Are Developing and Maintaining Secure Applications
How Enterprises Are Developing and Maintaining Secure Applications
The concept of application security is well known, but application security testing and remediation processes remain unbalanced. Most organizations are confident in their approach to AppSec, although others seem to have no approach at all. Read this report to find out more.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2019-20477
PUBLISHED: 2020-02-19
PyYAML 5.1 through 5.1.2 has insufficient restrictions on the load and load_all functions because of a class deserialization issue, e.g., Popen is a class in the subprocess module. NOTE: this issue exists because of an incomplete fix for CVE-2017-18342.
CVE-2019-20478
PUBLISHED: 2020-02-19
In ruamel.yaml through 0.16.7, the load method allows remote code execution if the application calls this method with an untrusted argument. In other words, this issue affects developers who are unaware of the need to use methods such as safe_load in these use cases.
CVE-2011-2054
PUBLISHED: 2020-02-19
A vulnerability in the Cisco ASA that could allow a remote attacker to successfully authenticate using the Cisco AnyConnect VPN client if the Secondary Authentication type is LDAP and the password is left blank, providing the primary credentials are correct. The vulnerabilities is due to improper in...
CVE-2015-0749
PUBLISHED: 2020-02-19
A vulnerability in Cisco Unified Communications Manager could allow an unauthenticated, remote attacker to conduct a cross-site scripting (XSS) attack on the affected software. The vulnerabilities is due to improper input validation of certain parameters passed to the affected software. An attacker ...
CVE-2015-9543
PUBLISHED: 2020-02-19
An issue was discovered in OpenStack Nova before 18.2.4, 19.x before 19.1.0, and 20.x before 20.1.0. It can leak consoleauth tokens into log files. An attacker with read access to the service's logs may obtain tokens used for console access. All Nova setups using novncproxy are affected. This is rel...