Big Data & The Security Skills Shortage

Finding a security analyst with the data discovery experience to combat modern threats is like searching for the mythical unicorn. The person does not exist

Peter Schlampp, VP of Products, Platfora

April 29, 2015

4 Min Read

Recent high-profile, high-impact data breaches across industries, including financial, healthcare, and retail, prove that today’s cybercriminals are adept at finding and fully exploiting even the smallest security gaps. Detection of their malicious activity often comes much too late – and at great cost for companies and their customers.

Not surprisingly, business leaders are starting to ask more of chief information security officers (CISOs) and other security operations personnel. They want assurance that the organization and its assets are protected. And they are, increasingly, looking for ways to leverage security analytics to strengthen cybersecurity. To do that, security analysts will need to bring three diverse skill sets to the table:

  1. Security domain expertise: The security analyst must understand security data, incident response, attack vectors, and more.

  2. Data science expertise: The security analyst needs advanced analytical skills, like using machine learning and predictive analytics algorithms, and should know how to prepare data for analysis.

  3. MapReduce/Spark/Storm/Hive/Pig expertise: The security analyst must be able to code a number of big data technologies designed to optimize analysis of petabytes of data.

While it would be ideal to find a security analyst who is proficient in all of these areas, I can tell you with confidence that, like the unicorn, this person does not exist. In fact, in the security space, its next to impossible to find security professionals with just one of these specialized, essential skill sets.

The challenges ahead
Because stealthy cyber-attacks can operate in networks undetected for weeks, months, or even longer, security analysts must be able to identify and analyze patterns that span lengthy time periods. They also must be able to visualize all of this data in way that helps them “connect the dots” and identify activity that’s out of the norm.

Another issue is the hundreds or thousands of security incident alerts organizations receive every day -- the vast majority of which are not malicious activity or targeted attacks. Differentiating between true, targeted attacks and non-malicious incidents is extremely difficult unless security analysts are armed with the skills and tools they need to make them entry-level data scientists.

When security analysts ask the right questions of big data, they can discover attack sequences and better understand the business impact of these events. To maintain that focus, security analytics supporting these investigative workflows must handle #2 and #3 on the list of criteria for the security analyst/unicorn: data science expertise and MapReduce/Spark/Storm/Hive/Pig expertise.

Security analysts equipped with these tools can better harness their security domain expertise. They can analyze security incidents, detect root causes, and unearth larger attacks before adversaries can exfiltrate high-value data. As a result, analysts looking to combat modern threats by taking advantage of big data, and data discovery, will need to acquire, or hone the following skills:

Identify the sequence of an attack: Security analysts need to analyze data surrounding incidents to identify anomalies and patterns that are not “normal.” By factoring in IT, user, and business application data as context, they can reach a conclusion on the impact of a security incident.

Ask many questions and get fast answers: Security analysts must conduct security investigations based on hypothesis and suspicion. They must ask as many questions as needed, receive fast responses, and quickly pivot their investigations based on those responses.

Derive insights on petabytes of data: In many organizations, security events and audit logs from IT, user, and business applications can amount to 10 terabytes (TB) of data per day. Given that a typical data breach timeline is 243 days, security analysts need to detect anomalies and patterns going back as far as 12 months – which requires them to analyze petabytes of data.

Translate security incidents into business impact: Security analysts need a centralized view of IT, user, business application data and security event data. Multi-structured data must coexist in a single repository, and be transformed and correlated so that the outcome of security investigations is about business impact.

The unicorn is a mythical creature, but a security analyst with deep security domain expertise is certainly not. When you support a skilled security analyst with security analytics on big data, your organization will be able to gain a complete picture of network and data security risks, and more quickly detect and mitigate advanced cyber-attacks.


About the Author(s)

Peter Schlampp

VP of Products, Platfora

An early member of the Platfora team, Peter leads the charge to put big data in users' hands in a beautiful and easy way. Peter Schlampp remembers playing with the keys of a TRS-80 when he was seven years old - and since then, he's been destined to build category-defining products that make businesses faster, safer, smarter, and more profitable. Before coming to Platfora, Peter was vice president of marketing and product management at Solera, where he built the first DVR for network traffic. At IronPort, where Peter was senior director of product management, he defined the world's first email security appliance, making the world safe from spam and viruses. He then became director and general manager of the Cisco/IronPort Development Center in Bangalore, India. Peter has a B.S. in Biology from Boston College.

Keep up with the latest cybersecurity threats, newly discovered vulnerabilities, data breach information, and emerging trends. Delivered daily or weekly right to your email inbox.

You May Also Like

More Insights