Surviving Security Alert Fatigue: 7 Tools and Techniques

It's an all-too-common problem for today's security teams: Alerts stream from a range of tools (sometimes misconfigured) and flood operations centers, forcing analysts to analyze and prioritize which ones deserve attention. Suffice to say, major problems arise when critical alerts slip through the cracks and lead to a security incident.

"One of the biggest drivers of alert fatigue is the fact that people are unsure or unconfident about the configuration that they have or the assets they have," says Dr. Richard Gold, head of security engineering at Digital Shadows. "What happens is you end up with a lot of alerts because people don't understand the nature of the problem, and they don't have time to."

Dr. Anton Chuvakin, head of solution strategy at Chronicle Security, takes it a step further: Many businesses are overwhelmed by alerts because they have never needed to handle them.

"I think a lot of organizations, until very recently, still weren't truly accepting of the fact they have to detect attacks and respond to incidents," he explains. Now, those that never had a security operations center or security team are adopting threat detection and are underprepared.

The proliferation of security tools is also contributing to the alert fatigue challenge, Chuvakin notes. "Today we have a dramatically wider scope of where we are looking for threats," he continues. "We have more stuff to monitor, and that leads alerts to increase as well." The most obvious risk of alert overload, of course, is companies could miss the most damaging attacks.

Security staff tasked with processing an unmanageable number of alerts will ultimately suffer from burnout and poor morale, security experts agree. What's more, overwhelmed employees may also be likely to simply shut off their tools.

It isn't the technology's fault, notes Chris Morales, head of security analytics at Vectra. "We don't have a detection problem – we have a prioritization problem," he explains. Any given person in a commercial security environment is tasked with multiple jobs: parsing data, writing scripts, knowing the ins and outs of cloud – and managing arrange of tech in their environment.

"The amount of data being pushed through corporate networks today is unlike anything we could have imagined 10 years ago," says Richard Henderson, head of global threat intelligence at LastLine. Organizations are struggling, and the onslaught of alerts is putting them at risk.

Here, security experts share their thoughts on the drivers and effects of alert fatigue, as well as the tools and techniques businesses can use to mitigate the problem. Which strategies have you used to combat alert overload? Which are effective? Feel free to share in the Comments section, below.

(Image: VadimGuzhva - stock.adobe.com)

Define Your Use Case

The most important thing companies can do is spend time understanding the problem, Digital Shadows' Gold says. If you're using an intrusion detection system (IDS), for example, and seeing a ton of alerts, you need to investigate what in your environment could be causing so many false positives? What does a true positive look like compared with a false positive? What is a real threat?

To lessen the flow of alerts, security teams should have use cases for detections, he continues. What are your greatest worries? If you fear credit card data exiting the environment, you can use the IDS to put rules in place for the specific thing you fear.

Detection use cases should represent what your business is concerned about, says Gold. High-fidelity use cases very tightly define your priorities. "You don't want to say, 'I'm concerned about any data leaving the organization,'" he explains. "You want to say, 'I'm concerned about data that looks like this.'"

Adds LastLine's Henderson: "You need to take a step back and spend some time thinking about exactly what it is you're trying to protect. Are you just checking a box? Are you checking a list against some regulation, or rule, or vertical body that says we have to?" If so, you're not checking the right thing, he says. Step back and consider all of your data – do you know where it is?

Before you can trust an alert, you have to know your data, says ReliaQuest CEO Brian Murphy. Security teams should "leverage automation only after they can trust the data they're automating," he explains, emphasizing the importance of analyzing accurate information.

(Image: Pathdoc - stock.adobe.com)

Configure with Care

When you buy a threat detection tool, security information and event management (SIEM) system, or other platform, don't assume it will surface valuable information from the start, says Chronicle Security's Chuvakin. Companies can worsen alert fatigue with high expectations of systems that immediately generate streams of crisp, actionable alerts.

"I've never seen that," he notes. "Moreover, if you use a tool that [shows] every alert is crisp and actionable, you are probably missing a lot of things."

Misconfiguration is a common and dangerous problem, Digital Shadows' Gold says. Some alert systems are trigger-happy because security teams prefer to err on the side of caution rather than miss something. This leads to a "boy who cried wolf" scenario in which they are burdened with alerts that don't convey any value, so teams ignore them and don't respond if it's something important.

"It's often the case that people who are made to configure these things are not given the necessary training and background to configure them correctly," he says. Some systems may raise an urgent alert for a bug that is critical for that specific tool but difficult to exploit and may not demand a "stop the press" attitude. Businesses need to better understand their problems.

"Misconfiguration is the biggest issue we deal with today," Vectra's Morales says. "It creates more work for the security team."

(Image: Ronstik - stock.adobe.com)

Threat Tools: SOAR, SIEM, EDR

When people have an issue, it's typically related to one of two types of tools, Chronicle Security's Chuvakin says, citing his experience as a Gartner vice president and analyst. One of these is security orchestration, automation, and response (SOAR) platforms; the other is more modern tools like endpoint detection and response (EDR), traffic analysis systems, and some new SIEM tools designed to collect more contact around an alert.

While the SOAR route "is definitely something that's helping," he says, the security team may have to adjust how it would confirm alerts specific to the organization. A company running Active Directory may run a different playbook from a business using a proprietary system, Chuvakin says. If used correctly, a SOAR tool can take steps a human would take to qualify alerts.

EDR tools help get information into the hands of a human who is triaging an alert, he continues, and can provide a history of alerts on the same system in the past. "This, to me, is more about enabling the human to make a judgment call without too much digging," he continues.

Some companies outsource alerts to a managed service provider. In these cases, Chuvakin advises being specific in telling the MSP what to flag and what not to flag. Outsourcing, as well as SOAR and EDR, are all "part of the solution," he says.

"The challenge and best practice is for each organization to find its balance between automation of workflow, outsourcing, and then presenting things to humans," he says.

(Image: Gorodenkoff – stock.adobe.com)

Learn to Operate the Tools You Have

In some cases, an organization's budget for a tool only accounts for the tool itself and not the training needed to use it, Digital Shadows' Gold says. The training some companies offer can be expensive, he adds, but it makes sense compared with the cost of buying a tool and not getting value out of it.

Like misconfiguration, lack of training for personnel is an issue when it comes to alert fatigue, he says. Security teams need training, experience, and colleagues to learn from.

For large companies, LastLine's Henderson recommends adopting the same tool across multiple security teams. He explains the story of one customer, a financial institution, that deployed a product across the organization and later found another branch was also using it. Cross-collaboration among the teams led to improved integration and cost savings, Henderson says.

"If you share institutional knowledge, all of a sudden that product gets used a lot better," he adds.

(Image: Pixel-Shot - stock.adobe.com)

Be Sure Assets Are Up to Date

Ensuring assets are up to date in any given system, including EDR, VPN, firewall, and cloud, can help cut down on invaluable alerts as well, Digital Shadows' Gold says. Make sure the data being analyzed is as current as it can be, and have regular views of that information, he adds. It's easy for data to get stale.

You don't want to learn this the hard way, Gold adds. The security team may get an alert and, upon investigation, see the IP address in question doesn't exist. Something may still be running when it should have been turned off, or a new employee's laptop may not have been added to the asset registry. When things like this aren't updated, it can lead to unnecessary alerts.

"Technology needs to be maintained and cleaned up," ReliaQuest's Murphy says. "You need to go through and make sure the rules, alerts, and dashboards that were built previously still apply to today's enterprise environment." Further, when you buy new tools, you need to be sure you're not buying a platform and/or capability you already have but aren't properly using.

(Image: Selinofoto - stock.adobe.com)

Account for 'Legit' Alerts

It's essential to coordinate with your security team and service providers to ensure alerts are not generated as a result of someone simply doing their job, Digital Shadows' Gold says. Many companies, especially those in the security space, often have to investigate malicious domains, files, or other suspicious findings. Security systems may flag this type of activity as an alert.

"If you're a security team inside a company that has to investigate security incidents, because of alerts raised by your investigation of security incidents, a lot of unnecessary drama can be created," he says. Someone may be flagged for using the Tor browser, for example, but have a legitimate business reason for doing research on it.

(Image: Elen31 - stock.adobe.com)

Learn from Past Mistakes

When something goes wrong, an important alert is missed, or an incident occurs, ReliaQuest's Murphy advises recording the details. "When something happens or there's a technical challenge, it's taking the time as a team to write down what happened," he explains. "It's an easy way to show people how things behave in the environment."

As you configure your systems and get a handle on alerts, it's helpful for current and future employees to measure success and improvement over time, Murphy says. Case studies can accurately portray how data flows throughout the business and how the team solves problems.

Security operations is a proactive process, Vectra's Morales explains, and the answer comes down to the company's risk awareness. Most companies are reactive toward security issues, when they should be coming in every day and identifying potential problems.

"You need to document, and map out, and take this seriously in a proactive way," he says.

(Image: Sfio Cracho - stock.adobe.com)