It's probably one of the more annoying parts of the job — those persistent pings alerting you that something may be wrong. Fortunately, there are best practices that IT professionals can implement to make security notifications less painful. This means identifying the greatest threats to your organization, selecting the right tools for alerts, and implementing a layered security approach to fend off the threat actors and minimize risk.
Turn Off the Alarms You Don't Need
The first line of defense against alert fatigue is to turn off the alarms you don't need, although choosing which notifications not to receive can be a nerve-racking endeavor. One of the most straightforward methods is to review alert logs and shut off notifications that have proven over time to be false alarms or false positives.
Consider How Quickly Intervention Is Needed
You can thank the Google site reliability engineers (SREs) for the next strategy. SRE teams — responsible for tasks like monitoring, emergency response, and capacity planning — have an alert/ticketing/log system in place to minimize alert overload by assigning a response to an event that is based on how quickly technician intervention is required. The three possible courses of action are:
- Alert: If a person must immediately intervene, an alert is sent.
- Ticket: If an event requires action, but can wait for normal business hours, a ticket is submitted.
- Log: If no action is needed, the event is logged for diagnostics.
Make Use of Smart Alarms
It's happened to most of us: Your phone blows up with notifications at 3 a.m., and you spend an hour trying to figure out and fix the problem. It doesn't have to be this way. Smart alarms not only alert you to a problem, but also can suggest ways to fix it by identifying the root cause. These types of alarms also provide historical data about events, enabling you to understand what happened immediately before and after a particular alert was triggered.
Prioritize Alerts and Set Rules to Detect Urgent Issues
All alerts are not created equal, so it's important to configure your performance monitoring tools so they send alerts only for the most critical events. By prioritizing alerts based on severity level, you can eliminate some of the noise generated by nonthreatening event notifications. Focus instead on setting up alerts for issues that can cause your servers to go offline, severely corrupt data, or result in significant data loss.
You can also manage alerts by applying specific thresholds and rules. When you define performance thresholds, you're not notified until a value for a specified metric reaches a concerning level — for example, when free disk space or free physical memory levels are dangerously low. This frees up technicians' time because they aren't constantly monitoring metrics. Setting rules for alerts also allows you to customize actions, such as how frequently you want to be notified.
The Importance of Layered Security Architecture
For in-depth defense, it's important to have a layered approach to security that encompasses physical, technical, and administrative controls. Physical controls prevent physical access to IT systems, such as locked doors. Technical controls protect network systems or resources using specialized hardware or software, such as firewall appliances or antivirus programs. Administrative controls consist of policies or procedures directed at employees — for example, instructing users to label sensitive information as confidential.
You also should incorporate security layers to protect individual facets of a network. Access measures include authentication controls, biometrics, timed access, and VPNs. Workstation defenses incorporate antivirus and anti-spam software, while data protection looks at data-at-rest encryption, hashing, secure data transmission, and encrypted backups. Firewalls and intrusion detection and prevention systems fall under perimeter defenses. And, finally, monitoring and prevention addresses logging and auditing network activity, vulnerability scanners, sandboxing, and security awareness training.
When looking for solutions, opt for ones that provide regular vulnerability scanning, monitor for compromised credentials, and offer employee security awareness training. And, to rest soundly at night, have a business continuity and disaster recovery (BCDR) tool in place. This way, if the worst-case scenario does happen, you'll be able to recover your organization's data and resume regular business operations.