20 Questions Security Leaders Need To Ask About Analytics

What problem(s) are you trying to solve? This may seem like an obvious question, but have you ever tried asking a potential vendor what problem or problems they’re trying to solve? Perhaps you will receive a clear, concise, and straightforward answer. Or, perhaps you will receive an answer that will leave you wondering if this particular vendor has any real-word understanding, operational experience, and/or deployments. 

If you get an answer that make sense, does it describe a problem you are looking to solve?

What data do vendors operate on? Even the greatest analytics and algorithms in the world need data to operate on; not just any data of course, but the specific data that the various analytics and algorithms were designed to work with. 

Does the vendor you’re considering leverage data that you have readily available or can easily collect? 

How difficult is it to get that data to the vendor for processing and analysis? 

What additional bandwidth usage will you incur moving data around? 

What additional cost requirements will you encounter when looking to retain the right amount of data to produce the desired results? 

Will the solution be able to scale to the volume of data you need?

What is the signal-to-noise ratio? 

What are the costs/benefits in terms of your security organization’s efficiency, effectiveness, and workflow? 

What is the cost of polluting the workflow with a large number of low fidelity, nonactionable, noisy alerts? This noise adds to the organization’s daily workload while simultaneously detracting from its ability to focus on the signal it needs to focus on.  Analytics often promise the benefit of detecting the previously undetectable. In reality, that benefit varies by solution. 

What is the cost in terms of efficiency and resource-allocation of introducing a ton of noise into the environment versus the potential benefit of additional detections that an analytics solution may provide? Additional detection capabilities can quickly get washed away by a sea of false positives.

How does the solution integrate into your workflow? 

How open are you to introducing another tool or layer of complexity into your security workflow? 

Do you need an analytics solution to integrate seamlessly into your workflow without requiring the team to learn additional skills or review additional consoles? 

What amount of overhead in terms of people, process, and technology does an analytics solution require in order to function properly? 

How complex is the solution to deploy and how much customization is required to get the solution up and running?

What methodologies does the approach use? Lots of people like to talk about data mining and machine learning when they talk about analytics. But does a potential vendor really leverage data mining and machine learning? The dirty little secret in the analytics field is that while many solutions talk about data mining and machine learning, some of them rely on signatures and triggers behind the scenes.

What will I get out of an analytics solution on a daily basis? 

Will it provide me with additional detection events of interest or high fidelity jumping off points for hunting?